xen dev blog

I'll be making regular updates of what I'm doing in this thread for anyone who cares.

Currently loading test data onto an 8 node Percona cluster. Percona is a high performance variant of mariadb cluster.

Loading in the IP audit table which is 13 million rows and ~1.2Gib on MyISAM. To avoid running joins against this massive table the username of each entry (which is otherwise just userid, timestamp, IP and post/pmid) is encoded directly in the table which is why its so big. Further, char rather than varchar is used to make the table lookup really fast (fixed length row means instant indexing). This table is used in production as a lookup table when signing up users so it has to be quick.

The table has to be converted to InnoDB (or Percona's variant of it) to be loaded into the cluster. So far about 75% loaded and taking 2.1Gib on each node. To be expected, InnoDB takes more space.

Getting some 'failure to report' messages regarding committing every so often but the data appears to be writing successfully.

 

Good news, the percona cluster took the test data without incident. An incomplete import of the site (converted for xen) is now running on the new cluster with a disk footprint of about 18gb. Expected final footprint to be about twice that.

Will do extensive testing tomorrow to ensure everything is going to work smoothly with this cluster solution before moving onto the remaining non-database work.

 

I will be giving CDT early entry so they can help with transition

 

Spending quite a bit of time today debugging xen installation on the cluster