Frogboy

Join Date 03/2001

+2419

Libraries: Status Report

So there are 3 variables in running a vastly popular website:

1) Bandwidth
2) SQL server performance
3) Scripting performance

Bandwidth is the easiest to solve. You just pay more money and you get more bandwidth. But on a site as popular as WinCustomize, the number of people on at the same time (think thousands) require a lot of SQL and page generation resources.

So even as we increase the bandwidth, you then end up where the SQL server gets maxed. So then you increase your SQL server resources and then more people are able to get on until the scripting resources are maxed. That's where we are now.

You can optimize things, tweak things, etc. And you're always battling out having a site that looks nice and presents information and isn't too hard to code and maintain with the demands of thousands of users browsing the libraries at the same time.

So what got this going in the first place? First, as some long time users know, late last Fall we increased our servers and bandwidith and everything else and our traffic increased dramatically. By March, we'd reached a new level of website popularity (Which is good except that since we're not ad driven, it's a mixed blessing). But we also just launched GalCiv.com which is seeing traffic that is an order of magnitude beyond what we had projected.

We have the bandwidth situation reasonably okay, but the SQL and scripting resources are what we have to work on further.

Locked Post

Advanced Search

Watch this post

Do not email me updates for this post Email me updates for this post

Successfully updated karma reason!

Woohoo! My favourite site is in working order again.
You'll never know how much I appreciate this site. I just wish I could afford to subscribe or had artistic talent so that I could contribute in a real way.

Anyhoooo... ramble over.
Thanks!

Successfully updated karma reason!

I don't want to start a chaos here but what about using php+apache+mysql? Yahoo! just moved from they own server side language to php.

It's just an idea...

Successfully updated karma reason!

Glad things are getting back slowly but surely.

Wallpaper is down? *Phew* For a minute I thought my wall was already ready for retirement because of the 0 downloads status.

Anyway, sounds like a lot of work; I commend you for the valiant efforts to keep this wonderful site running nicely for all of us!

I know it can be stressful trying to get so much imperative stuff done properly when you feel rushed but taking the time to do things right and make the necessary accommodations/changes will save headaches further down the line for everyone especially when there’s only so much blood you can get from a maxed out turnip.

I have all the patience in the world...but okay, that might not be a fair statement since I have a subscription TO OD etc. and therefore my access isn't/hasn't been hampered.

Va-Va-Varoom!

Successfully updated karma reason!

Sign Up or Login and this ad disappears!
There are many great features available to you once you register. Sign Up for a free account and browse the forums without ads.

Yes, PHP is definitely a wise way to go about things.

Also, with Apache + PHP + MySQL you can get special cache modules which work like so:

A page is generated dynamically. Yay! Now that page has been stored in the cache of the server so if it is called again with the same parameters and hasn't been changed, the cache is dished out and no parsing is required. Basically it just lightens the load on the server's processing end, but bandwidth usages stays the same.

Then you can use what's known as GZip output compression. This may or may not be viable on such a large scale because I'm pretty sure it takes some processing to compress the pages, which might create some lag. But it could be worth a try. What happens is the pages are compressed using GZip and sent the browser and they are much smaller in size. Any browsers capable of handling GZip-compressed pages (still HTML, with .php or .html extensions etc) will render the pages as normal but will have downloaded a significantly smaller amount of data.

PHP + Apache would certainly be a good way to go for security reasons, too

Successfully updated karma reason!

yes, get rid of the windows based servers!

Successfully updated karma reason!

Go ahead and flame me if I'm horribly wrong, but one solution would be to do what Kuro5hin.org does: the servers on the outside world serve static pages and route the database inserts and such things, and have multiple servers on an internal net that actually do all the hard DB work. Of course, it was a buttload of work for them to set up and would prolly be even harder for yall. Just a thought. Of course, they use a non-wind0ze server setup. Wait, this is a windows customization site... isn't preaching anti-windozeness some sort of heresy?

Successfully updated karma reason!

The bottom line, IMHO, is there is just no good substitute for FAST servers balancing a heavy SQL load.

Frogboy, have you guys seen these: http://www.cenatek.com/ - Solid state ramdisks designed for what you need, I think. They can be self-powered to retain data, etc. With some redundancy you could guarantee no data loss, and have the frontline servers running about as fast as is possible, considering the whole damn thing is in RAM.

It's worth a look, anyway...I have no affiliation, BTW, just one of the bookmarks I keep as a "when I get the money" link.

Successfully updated karma reason!

as i recall, the origional reason for this site being built using asp is that is what the developers are familar with

back in the begining, this site was conjoured up out of nothing in next to no time, so stardock went with what they new

Successfully updated karma reason!

Well I think the time has come that they will have to learn a new language out of necessity.

PHP is actually very easy to learn. Its syntax rules are very simple. The only think one needs to know is what functions are available to them. Functions can be found at the PHP site (php.net) so it's easy to find what you need.

Then there's JSP (Java Servlets), but I'm not sure how much faster/slow that would be than PHP or ASP. My guess would be faster, but it's definately not as easy to learn and write as PHP.

Successfully updated karma reason!

i have t o say great job brad, walls arnt up though... just uploaded a new wall, and bam library down, o well so goes my luck

Successfully updated karma reason!

i know i'll get bad looks from all the anti-microsoft people out there, but what about asp.net? my understanding is that pages in .net are pre-compiled, not interpreted, plus the language is full-blown visual basic (or visual C# or whatever) which is more capable but not too different than what the wincust people already know. i'm sure you've all thought of this already though...

Successfully updated karma reason!

asp.net is definately the way to go,

the way in which the asp.net is parsed and run by the server makes it not only very powerful, but also highly efficient. It does similar to the Apache thing mentioned above in that it stores pages in cache. asp.net allows for far higher simultaneous connections at lower system resource levels. PHP is in essence an old technology. asp.net software with good server load balancing can run the heaviest of sites. Switching to asp.net will also allow for less SQL calls in the way it deals with recordsets and thus lessening the demand on the SQL servers.

Good luck with the site, and keep up the good work, shame the Wall's are down as that's what I came here for today.

Successfully updated karma reason!

so does this mean that we can not view thumbnails inside of wincustomizer browser
(Skins an themes)

Successfully updated karma reason!

javascript:smilie('

Successfully updated karma reason!