xen dev blog

[Moes]

It's nice to see such detailed updates and I'm glad that you managed to get these problems solved so fast.

 

[Loyal 2 da game]

Cleaned up the vb3->xen conversion script somewhat today and automated loading the cluster with fresh data. This takes about 2 hours from fresh vbul data to completed xen data live on the cluster. This will be the minimum downtime window on changeover.

Trying to decide what to do about usernames with weird unicode crap in them. There are about 300 users with more than 20 posts each who are not banned and who have the world's most fucked characters in their usernames. It is possible to import these directly into xen but I'm not sure I really want to. I might end up sending these to a separate table and allowing (forcing) a username change to a visually similar normal username on login. Alternatively I could import them directly and grandfather them in (by disabling unicode usernames for new signups). Haven't decided yet.

Still having a significant problem with xenforo itself on the cluster. After testing, the cluster is working fine, however xen isn't. Logins and password recoveries currently failing (but succeeding with the same code on the non-cluster test server). Obvious culprit is the move from MyISAM to cluster-based InnoDB tables. This is blocking further progress at the moment. It's probably something relatively simple I will figure out tomorrow.

Word man my name has some pretty fucked characters in it, better add me to that separate kids table and force a name change on me back to loyal to da game. cheers

 

Word man my name has some pretty fucked characters in it, better add me to that separate kids table and force a name change on me back to loyal to da game. cheers

*bronco weenie

 

Ok I'm up

barely


Pingdom<->cloudflare watchdog script ran perfectly overnight. The site was switched to under attack mode for about 15 minutes at one point and then back to normal the rest of the time.


I finished writing the blob storage system for xen last night, allowing me to delete all those thousands of files. Unfortunately xen continues to generate many files onto the file system including sitemaps and other things. I will pull its write perms and see what crashes a bit later.

At the moment I am about to extend the DataWriter class in xen to force a write into the blob database on avatar/attachment upload. This is the small addon I mentioned earlier.

 

Ok I'm up

barely


Pingdom<->cloudflare watchdog script ran perfectly overnight. The site was switched to under attack mode for about 15 minutes at one point and then back to normal the rest of the time.


I finished writing the blob storage system for xen last night, allowing me to delete all those thousands of files. Unfortunately xen continues to generate many files onto the file system including sitemaps and other things. I will pull its write perms and see what crashes a bit later.

At the moment I am about to extend the DataWriter class in xen to force a write into the blob database on avatar/attachment upload. This is the small addon I mentioned earlier.

Do you have a degree or are you self taught?

 

Do you have a degree or are you self taught?

 

Why two degrees on different subjects as opposed to a higher degree in one subject? Four years could have put a PhD in computer science or something similar in your hands.

 

Why two degrees on different subjects as opposed to a higher degree in one subject? Four years could have put a PhD in computer science or something similar in your hands.

That's true but I prefer to do what interests me. A PhD in computer science sounds like the most boring thing in the world.

 

That's true but I prefer to do what interests me. A PhD in computer science sounds like the most boring thing in the world.

I met a guy while on an internship that had a PhD in particle physics. He was on a little over the average wage for the UK and worked in HR/PR for a bank. Apparently he found physics boring towards the end of his PhD and didn't have enough resources to specialise in something new and he had worked in a bank ever since. Something to consider if you ever feel like you missed out.

Xen is something I'm not familiar with. I'm assuming the move is partly for security, but are there other benefits behind the scenes? It seems like a lot of effort.

 

That's true but I prefer to do what interests me. A PhD in computer science sounds like the most boring thing in the world.

Exactly why I'm not doing it either l0l

PhD in physics could be kinda cool though

 

I met a guy while on an internship that had a PhD in particle physics. He was on a little over the average wage for the UK and worked in HR/PR for a bank. Apparently he found physics boring towards the end of his PhD and didn't have enough resources to specialise in something new and he had worked in a bank ever since. Something to consider if you ever feel like you missed out.

Xen is something I'm not familiar with. I'm assuming the move is partly for security, but are there other benefits behind the scenes? It seems like a lot of effort.

I'm guessing that a big part is for security and the rest is for scalability. VB3, what we're on now, is old as hell. To my knowledge it isn't receiving updates anymore except for maybe exploit patches, if that. With newer software the site isn't only more secure but will take an increase or decrease in scale with a lot less effort than dated software. I could be wrong though.

 

I'm guessing that a big part is for security and the rest is for scalability. VB3, what we're on now, is old as hell. To my knowledge it isn't receiving updates anymore except for maybe exploit patches, if that. With newer software the site isn't only more secure but will take an increase or decrease in scale with a lot less effort than dated software. I could be wrong though.

Actually it might surprise you to learn that vb3 is better at scaling than xenforo. That's why its so much work to changeover. Xenforo is designed for small communities. Only a few large communities have taken the plunge to change to it and only via a lot of hard work. I've been following the 2011 transition of digital point by their webmaster and they ran into most of the same problems I am.

In fact I bought their conversion script which forms the basis of our data import system. I've had to modify it substantially of course. It's written for an older version of xenforo and doesn't incorporate our modified version of vbulletin.

The main reasons we are moving to xenforo is: mobile support, better security (and not end of life, so constant updates) and numerous modern features which vb3 lacks.

I've found the function I need to replace with this addon. Keep getting distracted by other shite.

 

update on this:

xen's file writing runs pretty deep. My addon is going to need to override a whole bunch of its behaiour including how and where it writes temp files. I could just hack the code directly but this means we can't run regular updates, so instead I'm learning how to write Xen addons using best practices at the moment. It is significantly different to vbulletin. For starters they use a (proper) MVC system and secondly they do have hooks but they are only used to extend existing classes with your addon class.

Annnnd predictably one of the classes I need to extend has no code event listener to allow extension. Still researching.

 

I need to extend XenForo_Helper_File in order to begin curbing some of XenForo's file writing tendencies. Basically the entire software is heavily dependent on being able to modify the local file system (reading and writing files to disk). For very obvious security and scaling reasons I want to push all this functionality off to my blob database which lives on the cluster.

Unfortunately XenForo's addon system does not support extending helper classes. So now I either have to figure out how to make it support extending helper classes or patch the base php files themselves (i.e. hack the code rather than write an addon.) I will only do this last as a final resort because any XenForo update we apply later which updates that class will override our modifications.

 

[Pain]

Can we get animated UI's(ones that glow/flash etc) with this update(HIGH DONOR PERK PLS???????????????????)


I personally have so many things I want added that's on the do not suggest list/on the to-do list its scary lol.


Thanks for the great work though, I hope to see a lot of quality of life updates(more things we notice as members of sythe compared to backend stuff) this year I'm sure u have good plans!

 

[Shall Skill]

lol now you have a blog too? Need a fork and knife with that bro? Cuz ur biting off me hella

 

Because xenforo so extensively reads and writes to and from the file system and because it deliberately prevents you from modifying this behaviour by not providing addon hooks for the places these functions are used, I am now considering an orthogonal approach: A symbol table override. Effectively I'm looking at the possibility of wrapping the PHP file IO functions it uses so I can catch the ones I want (based on path). These would then be diverted to the blob database. I think this might be the fastest and most efficient way to do it. In other words: Effectively sandbox the IO of the application in a sandbox I control, without even modifying the application.

[root@sythep1 library]# grep -n -H -R "fopen" * | wc -l
106
[root@sythep1 library]# grep -n -H -R "file_put" * | wc -l
35
[root@sythep1 library]# grep -n -H -R "file_get" * | wc -l
50

As you see that's not even all the IO functions. This is what I would consider extensive file system use. A lot of those are fopens with +w (i.e. write) and they are just ordinary local files in the live serving directory, which is ridiculous if you ask me. I don't understand why the xenforo developers have done this. The only reason I can think of is that they were expecting all instances of their forum software to be communities with less than a thousand users.

Anyway I'm taking tomorrow off to go to the beach. I've been working 60+ hour weeks on this and my gf is going to kill me in my sleep if I don't take a break. Don't worry, I'll be back at it on Monday morning.

I might get a start on this tonight, i.e. investigate if the symbol table overrides are a practical possibility.

 

If you'd like a hand with any UI design, you know where I am. I have a good working knowledge of XenForo styles and a hankering to do something a little different from what I do at work..

Either way, all the best with getting XF going!

 

Can we get animated UI's(ones that glow/flash etc) with this update(HIGH DONOR PERK PLS???????????????????)


I personally have so many things I want added that's on the do not suggest list/on the to-do list its scary lol.


Thanks for the great work though, I hope to see a lot of quality of life updates(more things we notice as members of sythe compared to backend stuff) this year I'm sure u have good plans!

We had animated UI's once upon a time on vB.

 

If you'd like a hand with any UI design, you know where I am. I have a good working knowledge of XenForo styles and a hankering to do something a little different from what I do at work..

Either way, all the best with getting XF going!

By good knowledge of xenforo styles would you include a working understanding of CSS3/4?

Because most of what we need is CSS hacks