xen dev blog

Cleaned up the vb3->xen conversion script somewhat today and automated loading the cluster with fresh data. This takes about 2 hours from fresh vbul data to completed xen data live on the cluster. This will be the minimum downtime window on changeover.

Trying to decide what to do about usernames with weird unicode crap in them. There are about 300 users with more than 20 posts each who are not banned and who have the world's most fucked characters in their usernames. It is possible to import these directly into xen but I'm not sure I really want to. I might end up sending these to a separate table and allowing (forcing) a username change to a visually similar normal username on login. Alternatively I could import them directly and grandfather them in (by disabling unicode usernames for new signups). Haven't decided yet.

Still having a significant problem with xenforo itself on the cluster. After testing, the cluster is working fine, however xen isn't. Logins and password recoveries currently failing (but succeeding with the same code on the non-cluster test server). Obvious culprit is the move from MyISAM to cluster-based InnoDB tables. This is blocking further progress at the moment. It's probably something relatively simple I will figure out tomorrow.

 

Already got way too smashed at Christmas. Downed a slab of tallie cans in a couple of days + whiskey and other drinks. About a week's worth of hangover still wearing off. So, naturally, having a quiet new years.

Also update: I just had the non-cluster test server which is currently working for logins and password resets do a connection off to the cluster rather than its normal db. This reproduced the problems I've been having on the cluster. So it is definitely to do with xen--cluster incompatibility. Looking into it more.

 

Ok update on the login issues with xen loading on cluster:

Traced the problem to an incomplete import of the authentication table. SQL inserts for the authentication table are present in the dump being imported. I'm rerunning the import script by hand to find out what's happening. Possibly we are hitting some sort of cap or limit at that point in the import resulting in an incomplete import. Also possible that there is an SQL syntax incompatibility or similar between the dump and the receiving cluster.

 

Good news. Xen cluster login issues have been fixed. It was an import issue and not a xen-cluster incompatibility. Which is very good.

Worryingly the automated import script screwed this up twice. The import script uses the same commands I issue by hand. I am going to add a table row count check for future to ensure the imports are 100% complete. However it seems like it might be worthwhile just running the final import command by hand and keeping an eye on it to make sure it finishes completely.

Now I'll continue testing xen on the cluster and see if we are having any more issues.

 

Currently setting up http load balancer for the cluster and moving all development activity to the cluster.

 

Load balancer is configured and running.

Moving development stuff to the cluster nodes I realized xen only allows storage of avatars and attachments in the file system. No database storage is supported.

This is a problem for us. Firstly the file system is not designed to have hundreds of thousands of image files just hanging around. It makes searching slow, and it makes replication of the file system slow. Secondly I want to lock down all the nodes' file systems for security. Typically a good practice is to have a static set of files and directories that the site is served from which are owned by another account (dummy supervisor account) which doesn't give write access to the files or directories. The http daemon then runs on another account (service account) and is only permitted to read these files. This way even if an attacker finds an "upload exploit" in say php or the http daemon they will be unable to write to the file system and the exploit will do them no good.

This is part of a strategy called defensive programming ( https://en.wikipedia.org/wiki/Defensive_programming )

Obviously the way xen is configured at the moment it will need write access to the filesystem which I'm not willing to give it, so I will have to find another solution to avatar and attachment storage.

 

Digital point (another large forum who uses xen) keeps their avatars and attachments in a database via custom addon but they haven't released their code.

I haven't seen any up-to-date addons to allow for what we need here so I will probably write one myself. It's not very complicated so it shouldn't take more than a day. I'll make a small start tonight (setup the tables etc.) and write the addon proper tomorrow after some sleep.

 

Created the attachment & avatar blob database and access script. Writing an import script to load in the existing files now. Will add rewrite rules to redirect avatar and attachment requests to the access script next. The final thing needed is a very small xen addon which changes the upload success function from a move temp-file command to a blob insert. Probably about 3 or 4 lines. Hopefully the cluster doesn't kick up a fuss about storing a few gb worth of blobs. We'll see.

 

Here's my blob sql insertion generator for xen data

Test import of 10 blobs went fine. Now to import 100,000 of them. Fingers crossed.

Edit: just generating the inserts is taking its sweet time. Fuck xen for having a stupid avatar storage system.

 

Importing nicely!