Large Threads & Sythe Speed

[Annex]

Do you even know how much space 521k posts take up? Let's be EXTREMELY generous and say 1kb per post, that's ~521 mbs. That's almost nothing. Now add in that a single post is nowhere near 1 kb and the space required is next to none.

Old threads often generate traffic through google searches and I'm sure they've come in handy in staff investigations before, there is no real reason to remove them.

1kb is about 1000 characters, but the problem is that when you add the time of the post, the user posting, and all the stuff thats part of a query then it does add up. For example the quote BB code is 44 characters for this specific instance, which is about 4.4% of a kilobyte.

Anyways the process of mass querying is what slows the server down, realistically eventually (if he hasn't already) sythe is going to need to get a server that only runs mysql (or whatever sql server). Its not difficult to implement but it costs a lot of money to do so.

 

[iJava]

1kb is about 1000 characters, but the problem is that when you add the time of the post, the user posting, and all the stuff thats part of a query then it does add up. For example the quote BB code is 44 characters for this specific instance, which is about 4.4% of a kilobyte.

Anyways the process of mass querying is what slows the server down, realistically eventually (if he hasn't already) sythe is going to need to get a server that only runs mysql (or whatever sql server). Its not difficult to implement but it costs a lot of money to do so.

Quite right, a lot of other larger sites have a server for images, a sql server and a file server. This means that the strain is split across 3 different servers.

 

It's not a space issue, it's a ram issue from multipple people opening the large threads

No, it's an IO speed issue.

The server has plenty of ram and plenty of disk. The bottleneck (as in most computer systems) is pulling data from disk. On small threads this is simple. On large threads the database has to pull a lot of data in order to execute basic queries such as view last post and change page.

While it pulls the data it locks other queries from pulling their data. Hence the lag.

I've considerably increased both read and write caching, but as I've said repeatedly: Neither the forum software nor the database software is really designed with these mega queries in mind. They are efficient for small threads, and completely crap for large threads.

And FYI, we've already split out images and static files to other servers.

We basically need to rewrite the way vbulletin handles threads to fix the problem. (I.e. compile the front end of long threads into static content and store that alongside live content.)

When we move to vbul 4 I will also look into distributing the database with master/slave architecture which will help (but not fix) the problem.

 

[Shaaaaun]

No, it's an IO speed issue.

Sorry if this is a stupid question can somebody explain what 'IO'

 

[Annex]

Sorry if this is a stupid question can somebody explain what 'IO'

Input output, basically think of it as a rush hour LA trying to cram into a 4 lane highway. Same thing applies here, too many requests not enough bandwidth to support them all instantly. Unfortunately its not a fiber issue, but more of a SQL issue.

 

[Yenthe666]

I'm not quite sure, but couldn't some things be virtualized to 'overclock' the capabilities of the server?
I know big companys virtualize their system and thus can handle more data / capacity with the same hardware setup.

 

The issue is opening the large threads and searching them.

WE all suffered from the deletion/closure, but we suffered more when we couldnt open sythe cause of the load on the servers.

 

[SuF]

I'm not quite sure, but couldn't some things be virtualized to 'overclock' the capabilities of the server?
I know big companys virtualize their system and thus can handle more data / capacity with the same hardware setup.

You can only transfer stuff so fast. Nothing you can really do except split stuff up... Aka. Make more highways (that's a coding thing like Richard's saying).

 

Well I wasn't aware that the issue was not so much space as it was with just big threads. This suggestion can be closed/denied then, I understand more now.

 

[MohtasaUnique]

Archiving and permanently deleting threads is really at the discretion of Richard. I'd like to believe he acts with the best intentions and an idea of efficiency when it concerns the health of the forum.

Denied