xen dev blog

Fixed a bunch of display issues including the navigation crumbs. Had to add drop perms to the database user. The inability to truncate tables was blocking a whole bunch of xen's deferred cron backend stuff including rebuilding the template cache. I looked this up -- i.e. why can't xen just use delete from instead of drop? -- and turns out delete from is an allowed operation during a hot backup but truncate will block until the backup completes. So they've done this for data integrity, which is nice, but the downside is that the mysql user needs extra perms they wouldn't normally have for most applications.

Tomorrow I'm going to
1/fix the username situtation (i.e. decide what to do with unicode usernames)
2/fix or figure out unicode in thread names
3/make a start on importing dice and cards
4/look at url rewriting the old vb3 vbseo urls to the new xen urls.

early release to CDT and security team is getting pretty close, probably in the next couple of days I'll round up your IPs for the whitelist.

 

Added some optimization settings to the mysql nodes however this appears to have broken at least one server out of the cluster (desynced). Now I'm trying to figure out why it won't resync properly. Good to encounter and learn from these problems during testing rather than when the site is in production.

 

Looks like all the cluster members have to have (almost) identical configuration settings. That's quite interesting. Adding a larger key cache to just one caused it to desync and stay desynced. Now I'll try adding the optimisation to all of them simultaneously and run a rolling restart on the cluster. Either it will work or the whole thing will desync.

Essentially I am looking at speeding everything up a bit. I was doing speed metrics on page loads last night and it's not great. Granted I have debug shit turned on and I'm not doing opcode caching (yet) but I suspect most of the delay is from the database.

 

Lol now I'm learning that you should bring percona up one server at a time when rebooting the cluster. I just brought up 7 nodes after bootstrapping a single node and the 7 are aggressively crowding eachother for a state transfer from the bootstrap node.

 

Putting a bit of time into speed optimization today. I'll get stuck into the list I posted above afterwards. I want to see if I can get memcached sharding working because then I can store php sessions in it.

 

Replacing libmemcache with libmemcached on the php instances. This apparently allows me to configure memcache sharding.

 

package management system doesn't seem to have memcached configured for sasl so reinstalling manually

 

Some people will understand it, most won't.

Mostly I'm fed up with people assuming that running this site is simple or easy or cheap. This development blog can give some idea of the work I do on a day to day basis to keep this site online.

Also there's the off chance that someone is more knowledgeable about a specific thing I'm trying to do and they might give a pointer which will help. I'm always open to this possibility.