xen dev blog

[Sythe]

Cleaned up the vb3->xen conversion script somewhat today and automated loading the cluster with fresh data. This takes about 2 hours from fresh vbul data to completed xen data live on the cluster. This will be the minimum downtime window on changeover.

Trying to decide what to do about usernames with weird unicode crap in them. There are about 300 users with more than 20 posts each who are not banned and who have the world's most fucked characters in their usernames. It is possible to import these directly into xen but I'm not sure I really want to. I might end up sending these to a separate table and allowing (forcing) a username change to a visually similar normal username on login. Alternatively I could import them directly and grandfather them in (by disabling unicode usernames for new signups). Haven't decided yet.

Still having a significant problem with xenforo itself on the cluster. After testing, the cluster is working fine, however xen isn't. Logins and password recoveries currently failing (but succeeding with the same code on the non-cluster test server). Obvious culprit is the move from MyISAM to cluster-based InnoDB tables. This is blocking further progress at the moment. It's probably something relatively simple I will figure out tomorrow.

 

[Pure]

Cleaned up the vb3->xen conversion script somewhat today and automated loading the cluster with fresh data. This takes about 2 hours from fresh vbul data to completed xen data live on the cluster. This will be the minimum downtime window on changeover.

Trying to decide what to do about usernames with weird unicode crap in them. There are about 300 users with more than 20 posts each who are not banned and who have the world's most fucked characters in their usernames. It is possible to import these directly into xen but I'm not sure I really want to. I might end up sending these to a separate table and allowing (forcing) a username change to a visually similar normal username on login. Alternatively I could import them directly and grandfather them in (by disabling unicode usernames for new signups). Haven't decided yet.

Still having a significant problem with xenforo itself on the cluster. After testing, the cluster is working fine, however xen isn't. Logins and password recoveries currently failing (but succeeding with the same code on the non-cluster test server). Obvious culprit is the move from MyISAM to cluster-based InnoDB tables. This is blocking further progress at the moment. It's probably something relatively simple I will figure out tomorrow.

Thanks for working so hard for us, even though I don't understand this CMS mumbo jumbo! Hope you had a good Christmas/planning something for New Years!

 

[Sythe]

Thanks for working so hard for us, even though I don't understand this CMS mumbo jumbo! Hope you had a good Christmas/planning something for New Years!

Already got way too smashed at Christmas. Downed a slab of tallie cans in a couple of days + whiskey and other drinks. About a week's worth of hangover still wearing off. So, naturally, having a quiet new years.

Also update: I just had the non-cluster test server which is currently working for logins and password resets do a connection off to the cluster rather than its normal db. This reproduced the problems I've been having on the cluster. So it is definitely to do with xen--cluster incompatibility. Looking into it more.

 

[Pure]

Already got way too smashed at Christmas. Downed a slab of tallie cans in a couple of days + whiskey and other drinks. About a week's worth of hangover still wearing off. So, naturally, having a quiet new years.

Also update: I just had the non-cluster test server which is currently working for logins and password resets do a connection off to the cluster rather than its normal db. This reproduced the problems I've been having on the cluster. So it is definitely to do with xen--cluster incompatibility. Looking into it more.

Glad to hear you've had a great time while you're working out this xf transfer. I'd say keep up your hard work, but you can't no-life any harder

 

[Brendan]

Already got way too smashed at Christmas. Downed a slab of tallie cans in a couple of days + whiskey and other drinks. About a week's worth of hangover still wearing off. So, naturally, having a quiet new years.

cuz if you don't stop drinking the hangover will never come

p.s. happy new year in 2 hours

 

[Sythe]

Ok update on the login issues with xen loading on cluster:

Traced the problem to an incomplete import of the authentication table. SQL inserts for the authentication table are present in the dump being imported. I'm rerunning the import script by hand to find out what's happening. Possibly we are hitting some sort of cap or limit at that point in the import resulting in an incomplete import. Also possible that there is an SQL syntax incompatibility or similar between the dump and the receiving cluster.

 

[Sythe]

Good news. Xen cluster login issues have been fixed. It was an import issue and not a xen-cluster incompatibility. Which is very good.

Worryingly the automated import script screwed this up twice. The import script uses the same commands I issue by hand. I am going to add a table row count check for future to ensure the imports are 100% complete. However it seems like it might be worthwhile just running the final import command by hand and keeping an eye on it to make sure it finishes completely.

Now I'll continue testing xen on the cluster and see if we are having any more issues.

 

[FireZ]

Thanks richard for the work you have put into this site and continue to put in

 

[Life2dmax]

Good news. Xen cluster login issues have been fixed. It was an import issue and not a xen-cluster incompatibility. Which is very good.

Worryingly the automated import script screwed this up twice. The import script uses the same commands I issue by hand. I am going to add a table row count check for future to ensure the imports are 100% complete. However it seems like it might be worthwhile just running the final import command by hand and keeping an eye on it to make sure it finishes completely.

Now I'll continue testing xen on the cluster and see if we are having any more issues.

Yea no idea what the fuck you're saying but thanks for doing it

 

[Brian]

Good news. Xen cluster login issues have been fixed. It was an import issue and not a xen-cluster incompatibility. Which is very good.

Worryingly the automated import script screwed this up twice. The import script uses the same commands I issue by hand. I am going to add a table row count check for future to ensure the imports are 100% complete. However it seems like it might be worthwhile just running the final import command by hand and keeping an eye on it to make sure it finishes completely.

Now I'll continue testing xen on the cluster and see if we are having any more issues.

Thank you for the ongoing updates, it's nice to see what is happening in real time. Do you think you are confident enough to make an accurate estimation of when it will be live for users?

 

[Blupig]

i'll import your cluster any day of the week baby

 

[Sythe]

Currently setting up http load balancer for the cluster and moving all development activity to the cluster.

 

[Sythe]

Load balancer is configured and running.

Moving development stuff to the cluster nodes I realized xen only allows storage of avatars and attachments in the file system. No database storage is supported.

This is a problem for us. Firstly the file system is not designed to have hundreds of thousands of image files just hanging around. It makes searching slow, and it makes replication of the file system slow. Secondly I want to lock down all the nodes' file systems for security. Typically a good practice is to have a static set of files and directories that the site is served from which are owned by another account (dummy supervisor account) which doesn't give write access to the files or directories. The http daemon then runs on another account (service account) and is only permitted to read these files. This way even if an attacker finds an "upload exploit" in say php or the http daemon they will be unable to write to the file system and the exploit will do them no good.

This is part of a strategy called defensive programming ( https://en.wikipedia.org/wiki/Defensive_programming )

Obviously the way xen is configured at the moment it will need write access to the filesystem which I'm not willing to give it, so I will have to find another solution to avatar and attachment storage.

 

[Mootrucks]

idk wtf is going on but the frog fucks good

 

[Sythe]

Digital point (another large forum who uses xen) keeps their avatars and attachments in a database via custom addon but they haven't released their code.

I haven't seen any up-to-date addons to allow for what we need here so I will probably write one myself. It's not very complicated so it shouldn't take more than a day. I'll make a small start tonight (setup the tables etc.) and write the addon proper tomorrow after some sleep.

 

[Mootrucks]

Digital point (another large forum who uses xen) keeps their avatars and attachments in a database via custom addon but they haven't released their code.

I haven't seen any up-to-date addons to allow for what we need here so I will probably write one myself. It's not very complicated so it shouldn't take more than a day. I'll make a small start tonight (setup the tables etc.) and write the addon proper tomorrow after some sleep.

Sleep well master.

 

[Sythe]

Created the attachment & avatar blob database and access script. Writing an import script to load in the existing files now. Will add rewrite rules to redirect avatar and attachment requests to the access script next. The final thing needed is a very small xen addon which changes the upload success function from a move temp-file command to a blob insert. Probably about 3 or 4 lines. Hopefully the cluster doesn't kick up a fuss about storing a few gb worth of blobs. We'll see.

 

[Sythe]

Here's my blob sql insertion generator for xen data

find data | grep "\." | grep -v "index.html" | awk '{ system("echo INSERT INTO xen_data \\(file_path, file_timestamp, file_mimetype, file_name, file_data\\) VALUES \\(\\\""$0"\\\", `stat -c %Y "$0"`, \\\"`file --mime-type "$0" | cut -d\" \" -f2`\\\", \\\"\\\", LOAD_FILE\\(\\\"`pwd`/"$0"\\\"\\)\\)\\;");}'

Test import of 10 blobs went fine. Now to import 100,000 of them. Fingers crossed.

Edit: just generating the inserts is taking its sweet time. Fuck xen for having a stupid avatar storage system.

 

[Sythe]

Importing nicely!

 

[Brendan]

yaay!