xen dev blog

Ok I'm up

barely


Pingdom<->cloudflare watchdog script ran perfectly overnight. The site was switched to under attack mode for about 15 minutes at one point and then back to normal the rest of the time.


I finished writing the blob storage system for xen last night, allowing me to delete all those thousands of files. Unfortunately xen continues to generate many files onto the file system including sitemaps and other things. I will pull its write perms and see what crashes a bit later.

At the moment I am about to extend the DataWriter class in xen to force a write into the blob database on avatar/attachment upload. This is the small addon I mentioned earlier.

 

That's true but I prefer to do what interests me. A PhD in computer science sounds like the most boring thing in the world.

 

Actually it might surprise you to learn that vb3 is better at scaling than xenforo. That's why its so much work to changeover. Xenforo is designed for small communities. Only a few large communities have taken the plunge to change to it and only via a lot of hard work. I've been following the 2011 transition of digital point by their webmaster and they ran into most of the same problems I am.

In fact I bought their conversion script which forms the basis of our data import system. I've had to modify it substantially of course. It's written for an older version of xenforo and doesn't incorporate our modified version of vbulletin.

The main reasons we are moving to xenforo is: mobile support, better security (and not end of life, so constant updates) and numerous modern features which vb3 lacks.

I've found the function I need to replace with this addon. Keep getting distracted by other shite.

 

update on this:

xen's file writing runs pretty deep. My addon is going to need to override a whole bunch of its behaiour including how and where it writes temp files. I could just hack the code directly but this means we can't run regular updates, so instead I'm learning how to write Xen addons using best practices at the moment. It is significantly different to vbulletin. For starters they use a (proper) MVC system and secondly they do have hooks but they are only used to extend existing classes with your addon class.

Annnnd predictably one of the classes I need to extend has no code event listener to allow extension. Still researching.

 

I need to extend XenForo_Helper_File in order to begin curbing some of XenForo's file writing tendencies. Basically the entire software is heavily dependent on being able to modify the local file system (reading and writing files to disk). For very obvious security and scaling reasons I want to push all this functionality off to my blob database which lives on the cluster.

Unfortunately XenForo's addon system does not support extending helper classes. So now I either have to figure out how to make it support extending helper classes or patch the base php files themselves (i.e. hack the code rather than write an addon.) I will only do this last as a final resort because any XenForo update we apply later which updates that class will override our modifications.

 

Because xenforo so extensively reads and writes to and from the file system and because it deliberately prevents you from modifying this behaviour by not providing addon hooks for the places these functions are used, I am now considering an orthogonal approach: A symbol table override. Effectively I'm looking at the possibility of wrapping the PHP file IO functions it uses so I can catch the ones I want (based on path). These would then be diverted to the blob database. I think this might be the fastest and most efficient way to do it. In other words: Effectively sandbox the IO of the application in a sandbox I control, without even modifying the application.

As you see that's not even all the IO functions. This is what I would consider extensive file system use. A lot of those are fopens with +w (i.e. write) and they are just ordinary local files in the live serving directory, which is ridiculous if you ask me. I don't understand why the xenforo developers have done this. The only reason I can think of is that they were expecting all instances of their forum software to be communities with less than a thousand users.

Anyway I'm taking tomorrow off to go to the beach. I've been working 60+ hour weeks on this and my gf is going to kill me in my sleep if I don't take a break. Don't worry, I'll be back at it on Monday morning.

I might get a start on this tonight, i.e. investigate if the symbol table overrides are a practical possibility.

 

By good knowledge of xenforo styles would you include a working understanding of CSS3/4?

Because most of what we need is CSS hacks