What's wrong with the servers?

[iJava]

There are plenty of decent MySQL hosts out there and can't be too hard to find. Does mysql.com still offer hosting?

 

[Brendan]

I know I'm going to sound like a complete noob here, but can someone give an explanation in less technical terms on what exactly it is that causes all of these errors and server downtime? I know next to nothing regarding hosting :/

And I don't know whether it has something to do with my timezone, but are the errors really that bad? Between 3PM-12AM GMT +10 (The only time I'm really free to go online) I hardly ever get an error. And when I do, all I need to do is go back a page and click reply again.

 

[jakeyolo]

I think its just overflowing of the server

 

[Grave]

Sythe.org has 500 users in the last five minutes all accessing the same table of the same database, trying to read and write to it.

If you don't understand the difference between database concurrency issues, hardware IO issues and reverse proxy/caching issues then how can you possibly help?

To clarify: We only have the first two problems, one because of the way vbulletin is structured, the other because our host has been rubbish over the past few months. We do not have the third problem -- which is the only problem for which you have proposed a solution.

If you want to do some research into how hard it is to run large sites smoothly on mysql go look at facebook: http://www.facebook.com/MySQLatFacebook
They have a dedicated research staff to keep their mysql instances up.

Anyway long story short, saying our set-up is shit doesn't help fix any of the problems. I've already addressed the memory errors and I'm keeping a closer eye on them, with a script to inform me when the site is giving out 500 errors. The host has fixed its disk and IO problems.

I'm still looking into migrating us to a cloud hosted mysql solution, but that can't be rushed.

Have you already looked into services by Amazon or Rackspace? The latter seems to be especially reliable, as they have a managed hybrid cloud service.

Well again, the main issues are database contention.

Regarding upgrades... This is a common mistake among web masters (including professionals.)

Upgrading server software to the latest -- for no better reason than because it is the latest -- is usually a bad idea. Indeed, the latest versions of things are the least tested and tend to be the most vulnerable security-wise. It is generally much preferable to run servers on backdated versions of things that have already been through years of production exposure and have been patched and repatched repeatedly to get out all the flaws. Upgrading should be a last resort and should only be between major versions (i.e. vb3 -> vb4) and never at the first release of a major version.

That said, I will do a review when the time comes.

Just my little input on this: Correct, when a new version comes out, it's never a good idea to just blindly switch over when you are a large website. However, vBulletin 4 was released in 2009, and we are up to 4.2 now. By this point, most of the security vulnerability should be patched, and we've already gone past the point of zero support for the 3.6.4 version we currently have, which, by the way, also had some vulnerabilities that were patched in future versions.

And here are the Pros and Cons as I see them:

• vBulletin 4 has much better SEO, increasing the potential for the community to grow
• vBulletin 3 is unsupported, and we are stuck on the 2006 version
• Our version has 6 known vulnerabilities, which was taken down to 1 if we upgrade to 3.6.8
• If anything, we should at least upgrade to the end of the line in the 3 series, which was the most stable/secure version when Jelsoft abandoned it
• Since vB4 is supported, any security issues created by updates are patched quickly, or you can just stick with an older versions with no flaws
• Going by general logic, we can assume that the company you trusted enough by using their software knows how to handle their software, and improves it rather than degrading it and making it less secure
• Some other improvements that were made:
  
  • Improved search performance (no more disabling search)
  • Generally much better PM system
  • Better user security

Keep in mind that we can always go with IPB3, which I still see as better software performance and security-wise. Many large websites use this without any issues, so I don't see how Sythe.org cannot. I believe Powerbot also migrated without any severe issue. Also, not to be a complainer, but I don't see Powerbot going down daily either, and their database doesn't get leaked every year or two.

 

[kill dank]

Sounds like you're stooging for them.

Basically, yeah. Everything else was just my version of a legal disclaimer.

 

[Sythe]

And here are the Pros and Cons as I see them:

• vBulletin 4 has much better SEO, increasing the potential for the community to grow
• vBulletin 3 is unsupported, and we are stuck on the 2006 version
• Our version has 6 known vulnerabilities, which was taken down to 1 if we upgrade to 3.6.8
• If anything, we should at least upgrade to the end of the line in the 3 series, which was the most stable/secure version when Jelsoft abandoned it
• Since vB4 is supported, any security issues created by updates are patched quickly, or you can just stick with an older versions with no flaws
• Going by general logic, we can assume that the company you trusted enough by using their software knows how to handle their software, and improves it rather than degrading it and making it less secure
• Some other improvements that were made:
  
  • Improved search performance (no more disabling search)
  • Generally much better PM system
  • Better user security

Firstly we are already upgrading to vb4. That is not in debate. The only issue for vb4 is the look and feel and migrating custom code and plugins correctly.

However keep in mind that vb4 is even more database intensive than vb3 and will only exacerbate any contention issues -- which should be addressed first.

Secondly we do not run 3.6.4.

 

[SuF]

Firstly we are already upgrading to vb4. That is not in debate. The only issue for vb4 is the look and feel and migrating custom code and plugins correctly.

However keep in mind that vb4 is even more database intensive than vb3 and will only exacerbate any contention issues -- which should be addressed first.

Secondly we do not run 3.6.4.

It just says we do:

so if we don't that's where the confusion arose from.

 

[Sythe]

It just says we do:

It certainly does say that.

 

[Brendan]

It certainly does say that.

Isn't it a heavily modified version of 3.6.4 that we run? (I think I read that somewhere once.)

 

[Sordeo123]

i don't see why this issue is still being talked about. Sythe clearly stated his path and his knowledge of the issue and is doing something about it. Don't make harsh decisions and just take it a step at a time (as sythe was saying).

 

[Annex]

Isn't it a heavily modified version of 3.6.4 that we run? (I think I read that somewhere once.)

Yes and as far as I know back when I was admin me and Sythe patched most of the exploits that were found in the version it states that is being run.

 

[Sythe]

I am in the process of trying to migrate us to amazon web services at the moment. Unfortunately upgrading will lead to still more downtime. Bear with us.

 

[Brendan]

I am in the process of trying to migrate us to amazon web services at the moment. Unfortunately upgrading will lead to still more downtime. Bear with us.

Thanks for letting us know In theory, should we be getting better uptime and less errors in the long run?

 

[Schismatic]

It appears to me that there was a small rollback. Confirm?

 

[Sythe]

Small rollback. We tried amazon web services (which took many hours to transfer the database to) for about 30 minutes. It is far slower than our existing setup, so I've reverted.

 

[The Black Tux]

Small rollback. We tried amazon web services (which took many hours to transfer the database to) for about 30 minutes. It is far slower than our existing setup, so I've reverted.

Very interesting. I've never tested Amazon but I'm somehow disappointed in them already.

Could the same test be performed in Rackspace?

http://www.rackspace.com/cloud/public/sites/

 

[Sythe]

Very interesting. I've never tested Amazon but I'm somehow disappointed in them already.

Could the same test be performed in Rackspace?

http://www.rackspace.com/cloud/public/sites/

I've had bad experience with rackspace previously. Not wanting to repeat.

Just found the source of the 500 errors. I asked the tech to recompile apache with mod_deflate a couple of days ago. WHM appears to have taken it upon itself to update php at the same time and override config files with defaults. Long story short some modules were being loaded incorrectly.

Unfortunately the 500 errors were happening already before this due to memory capping, and I didn't think to check the error logs to see if they were different until I realised the number of 500 errors had increased dramatically. Tech has apologized for not checking logs after build (which he definitely should have.)

I've negotiated a ram and IO speed increase. We already have additional ram, and they are looking into the latter now. I also spent half my weekend working on grafting our vb3 skin to vb4 (which turns out to be quite difficult.)

So there you have it for updates, progress and endless stress. Did I mention I have the flu, two assignments and a lab due tomorrow? And it is currently 3:50 am. And people say I don't do anything for this site...

 

[The Black Tux]

Gotta love Richard. Good luck to the techs and you fixing the last few things.

I'm eager to see VB4

:love:

 

[Grim_Reaper]

Thanks for all the work you've put into Sythe recently, Richard.

Glad to see a hard working site owner.

 

[Karl]

I've had bad experience with rackspace previously. Not wanting to repeat.

Just found the source of the 500 errors. I asked the tech to recompile apache with mod_deflate a couple of days ago. WHM appears to have taken it upon itself to update php at the same time and override config files with defaults. Long story short some modules were being loaded incorrectly.

Unfortunately the 500 errors were happening already before this due to memory capping, and I didn't think to check the error logs to see if they were different until I realised the number of 500 errors had increased dramatically. Tech has apologized for not checking logs after build (which he definitely should have.)

I've negotiated a ram and IO speed increase. We already have additional ram, and they are looking into the latter now. I also spent half my weekend working on grafting our vb3 skin to vb4 (which turns out to be quite difficult.)

So there you have it for updates, progress and endless stress. Did I mention I have the flu, two assignments and a lab due tomorrow? And it is currently 3:50 am. And people say I don't do anything for this site...

Sounds promising, I hope this is the end to those irritating errors. Stupid that those guys did that, and thats unfortunate, goodluck on the assignments!