Is it important?
You can invest a lot of time and money into server tuning and code rewriting according the guidelines you have just learned, but your code's performance will be really bad if you do not take into account the hardware demands, and do not wisely choose the operating system suited to your needs. While the tips below apply to any webserver, they are aimed at administrators of mod_perl-enabled webservers.
Choosing an Operating System
First let's talk about Operating Systems (OS). While I am personally a Linux devotee, I do not want to start yet another OS war. Assuming this, I will try to define what you should be looking for, then when you know what you want from your OS, go find it. Visit the Web sites of operating systems you are interested in. You can gauge user's opinions by searching relevant discussions in newsgroup and mailing list archives such as Deja - http://deja.com and eGroups - http://egroups.com . I will leave this fan research to the reader. But I would use Linux or something from the *BSD family.
Stability and Robustness
Probably the most desired features in an OS are stability and robustness. You are in an Internet business, which does not have normal working hours, like many conventional businesses you know about (9am to 5pm). You are open 24 hours a day. You cannot afford to be off-line, for your customers will go shop at another service like yours, unless you have a monopoly :) . If the OS of your choice crashes every day or so, I would throw it away, after doing a little investigation, for there might be a reason for a system crash. Like a runaway server that eats up all the memory and disk space, so you cannot blame the OS for that. Generally, people who use the OS for some time can tell you a lot about its stability.
Memory Management
You want an OS with a good memory management, some OSes are well known as memory hogs. The same code can use twice as much memory on one OS compared to another. If the size of the mod_perl process is 10Mb and you have tens of these running, it definitely adds up!
Memory Leakages
Some OSes and/or their libraries (like C runtime libraries) suffer from memory leaks. We cannot afford such systems, for we already know that a single mod_perl process sometimes serves thousands of requests before it terminates. So if a leak occurs on every request, our memory demands will be huge. Of course our code can be the cause of the memory leaks as well (check out the Apache::Leak module). Certainly, we can lower the number of requests to be served over the process' life, but that can degrade performance.
Sharing Memory
We want an OS with good memory sharing capabilities. As we have learnt, if we preload the modules and scripts at server startup, they are shared between the spawned children, at least for a part of a process' life span, since memory pages can become "dirty" and cease to be shared. This feature can save us a lot of memory!
Cost and Support
If we are in a big business we probably do not mind paying another $1000 for some fancy OS with bundled support. But if our resources are low, we will look for cheaper and free OSs. Free does not mean bad, it can be quite the opposite as we all either know from our own experience or have read about in the news. Free OSs could have and do have the best support we can find. It is very easy to understand - most of the people are not rich and will try to use a cheaper or free OS first if it does the work for them. Since it really fits their needs, many people keep using it and eventually know it well enough to be able to provide support for others in trouble. Why would they do this for free? For the spirit of the first days of the Internet, when there was no commercial Internet and people helped each other, because someone helped them in first place. I was there, I was touched by that spirit and I will do anything to keep that spirit alive.
But, let's get back to our world. We are living in material world, and our bosses pay us to keep the systems running. So if we feel that we cannot provide the support ourselves and we do not trust the available free resources, we must pay for an OS backed by a company, and blame them for any problem. Our boss wants to be able to sue someone if the project has a problem caused by the external product that is being used in the project. If we buy a product and the company selling it, claims support, we have someone to sue. We do not have someone to sue other than getting ourself fired if we go with Open Source and it fails.
Also remember that if you spend less or zero money on OS and Software, you will be able to buy better and stronger hardware.
Discontinued products
You have invested a lot of time and money into developing some proprietary software that is bundled with the OS you were developing under. Like writing a mod_perl handler that takes advantage of some proprietary features of the OS and will not run on any other OS. Things are under control, the performance is great and you sing with happiness. But... one day the company who wrote our beloved OS goes bankrupt, which is not unlikely to happen nowadays, or they produce a newer incompatible version and they will not support the old version our code runs under (and this happens regularly). You are stuck with their last masterpiece and no support! What you are going to do then? Invest more into porting the software to another OS...
Everyone can be hit by this mini-disaster, so it is better to check the background of the company when making your choice, but still you never know what will happen tomorrow. The OSs in this hazard group are completely developed by single companies. Free OSs are probably less susceptible to this, for development is distributed between many companies and developers, so if a person who developed a really important part of the kernel lost interest in continuing, someone else will pick the falling flag and carry on. Of course if tomorrow some better project showed up, developers might migrate there and finally drop the development: but in practice people are given support on older versions and helped to migrate to current versions.
OS Releases
Actively developed OSs generally try to keep pace with the latest technology developments, and continually optimize the kernel and other parts of the OS to become better and faster. Nowadays, Internet and networking in general are the hottest targets for system developers. Sometimes a simple OS upgrade to the latest stable version, can save you an expensive hardware upgrade. Also, remember that when you buy new hardware, chances are that the latest software will make the most of it. Since the existing software (drivers) might support the brand new product because of its backwards compatibility with previous products of the same family, it might not reap all the benefits of the new features. It means that you could spend much less money for almost the same functionality if you were to buy a previous model of the same product.
Choosing Hardware
Since I am not fond of the idea of updating this section every time a new processor or memory type comes out, I will only hint about what you should look for and suggest that sometimes the most expensive machine is not the one which provides the best performance.
Your demands are based on many aspects and components. Let's discuss some of them.
In discussion course we use terms that may be unfamiliar to some readers:
Clustering - a group of machines connected together to perform one big or many small computational tasks in a reasonable time. Clustering is also used to provide 'fail-over' where if one machine fails its work is transparently transferred to another without any interruption of service.
Load balancing - users can remember only a name of one of your machines - namely of your server, but it cannot stand the heavy load, so you use a clustering approach, distributing the load over a number of machines. The central server, the one users access when they type the name of the service, works as a dispatcher, by redirecting requests to the rest of the machines, sometimes it also collects the results and returns them to the users. One of the advantages is that you can take one of the machines down for a repair or upgrade, and your service will still work - the main server will not dispatch the requests to the machine that was taken down. I will just say that there are many load balancing techniques. (See High-Availability Linux Project for more info.)
NIC - Network Interface Card.
RAM - Random Access Memory
RAID - META
Expected site traffic
If you are building a fan site, but want to amaze your friends with a mod_perl guest book, an old 486 machine could do it. If you are into a serious business, it is very important to build a scalable server, so if your service is successful and becomes popular, your server's traffic doubles every few days, you should be ready to add more resources dynamically. While we can define the webserver scalability more precisely, the important thing is to make sure that you can add more power to your webserver(s) without investing additional money into a software development (almost, you will need a software to connect your servers if you add more of them). It means that you should choose a hardware/OS that can talk to other machines and become a part of the cluster.
From the other hand if you prepare for a lot of traffic and buy a monster to do the work for you, what happens if your service does not prove to be as successful as you thought it would be. Then you've spent too much money and meanwhile there were a new faster processors and other hardware components have been released, so you lose again.
Wisdom and prophecy , that's all it takes :)
Cash
Everybody knows that Internet is a cash hole, what you throw in, hardly comes back. This is not always true, but there is a lot of wisdom in these words. While you have to invest money to build a decent service, it can be cheaper! You can spend as much as 10 times more money on a strong new machine, but get only a 10% improvement in performance. Remember that a four year old processor is still very powerful.
If you really need a lot of power do not think about a single strong machine (unless you have money to throw away), think about clustering and load balancing. You can probably buy 10 times more older but very cheap machines and have a 8 times more power, than purchasing only one single new machine. Why is that? Because as I mentioned before generally the performance improvement is marginal while the price is much higher. Because 10 machines will do faster disk I/O, than one single machine, even if the disk is much faster. Yes, you have more administration overhead, but there is a chance you will have it anyway, for in a short time the machine you have just invested in will not stand the load anyway and you will have to purchase more and think about how to implement load balancing and file system distribution.
Why I am so convinced? Facts! Look at the most used services on the Internet: search engines, email servers and the like -- most of them are using a clustering approach. While you may not always notice this, because they hide the real implementation behind proxy servers.
Internet Connection
You have the best hardware you can get, but the service is still crawling. Make sure you have a fast Internet connection. Not as fast as your ISP claims it to be, but fast as it should be. The ISP might have a very good connection to the Internet, but puts many clients on the same line. If these are heavy clients, your traffic will have to share the same line and the throughput will decline. Think about a dedicated connection and make sure it is truly dedicated. Trust the ISP but check it!
The idea of having a connection to The Internet is a little misleading. Many Web hosting and co-location companies have large amounts of bandwidth, but still have poor connectivity. The public exchanges, such as MAE-East and MAE-West, frequently become overloaded, yet many ISPs depend on these exchanges.
Private peering means that providers can exchange traffic much quicker.
Also, if your Web site is of global interest, check that the ISP has good global connectivity. If the Web site is going to be visited mostly by people in a certain country or region, your server should probably be located there.
And bad connectivity can directly influence your machine's performance. Here is a story, one of the developers told on the mod_perl mailing list:
What relationship has 10% packet loss on one upstream provider got
to do with machine memory ?
Yes.. a lot. For a nightmare week, the box was located downstream of
a provider who was struggling with some serious bandwidth problems
of his own... people were connecting to the site via this link, and
packet loss was such that retransmits and tcp stalls were keeping
httpd heavies around for much longer than normal.. instead of
blasting out the data at high or even modem speeds, they would be
stuck at 1k/sec or stalled out... people would press stop and
refresh, httpds would take 300 seconds to timeout on writes to
no-one.. it was a nightmare. Those problems didn't go away till I
moved the box to a place closer to some decent backbones.
Note that with a proxy, this only keeps a lightweight httpd tied up,
assuming the page is small enough to fit in the buffers. If you are
a busy internet site you always have some slow clients. This is a
difficult thing to simulate in benchmark testing, though.I/O performance
If your service is I/O bound (does a lot of read/write operations to disk, remember that relational databases are sitting on disk as well), you need a very fast disk. So you should not spend money on Video card and monitor (monochrome card and 14" B&W are perfectly adequate for a server -- you will probably be telnetted or ssh-ed in most of the time), but rather look for disks with the best price/performance ratio. Of course, ask around and avoid disks that have a reputation for headcrashes and other disasters.
With money in hand you should think about getting a RAID system. RAID is generally a box with many HDs. It is capable of reading and writing data much faster, and is protected against disk failures. It does this by duplicating the same data over a number of disks, so if one fails, the RAID controller detects it and the data is still correct on the duplicated disks. You must think about RAID or similar systems if you have an enormous data set to serve. (What is an enormous data set nowadays? Gigabytes, terabytes?).
Ok, we have a fast disk, what's next? You need a fast disk controller. So either you should use the one embedded on your motherboard or you should plug a controller card if the one you have onboard is not good enough.
Memory
How much RAM (Randomly Accessed Memory) do you need? Nowadays, chances are you will hear: "Memory is cheap, the more you buy the better". But how much is enough? The answer is pretty straightforward: "You do not want your machine to swap". When the CPU needs to write something into memory, but notices that it is already full, it takes the least frequently used memory pages and swaps them out. Swapping out means writing the data to disk. Another process then references some of its own data, which happens to be on one of the pages that has just been swapped out. The CPU, ever obliging, swaps it back in again, probably swapping out some other data that will be needed very shortly by another process. Carried to the extreme, the CPU and disk start to thrash hopelessly in circles, without getting any real work done. The less RAM there is, the more often this scenario arises. Worse, you can exhaust swap space as well, and then the troubles really start...
How do you make a decision? You know the highest rate at which your server expects to serve pages and how long it takes to do so. Now you can calculate how many server processes you need. Knowing the maximum size any of your servers can get, you know how much memory you need. You probably need less memory than you have calculated if your OS supports memory sharing and you know how to make best use of this feature (preloading the modules and scripts at server startup). Do not forget that other essential system processes need memory as well, so you should plan not only for the web server, but also take into account the other players. Remember that requests can be queued, so you can afford to let your client wait for a few moments until a server is available to serve it; your numbers will be more correct, since you generally do not have the highest load, but you should be ready to bear the peaks. So you need to reserve at least 20% of free memory for peak situations. Many sites have crashed a few moments after a big scoop about them was posted and an unexpected number of requests suddenly came in. (This is called the Slashdot effect, which was born at http://slashdot.org ) If you are about to announce something cool, be aware of the possible consequences.
Bottlenecks
The most important thing to understand is that you might use the most expensive components, but still get bad performance. Why? Let me introduce an annoying word: bottleneck.
A machine is an aggregate of many big and small components. Each one of them may be a bottleneck. If you have a fast processor but a small amount of RAM (memory), the processor will be under-utilized waiting for the kernel to swap the memory pages in and out, because memory is too small to hold the most used ones. If you have a lot of memory and a fast processor and a fast disk, but a slow controller - the performance will still be bad, and you will have wasted money.
Use a fast NIC (Network Interface Card) that does not create a bottleneck. If it is slow, the whole service is slow. This is the most important component, since webservers are much more network-bound than disk-bound!
Conclusion
To use your money optimally you have to understand the hardware very well, so you will know what to pick. Otherwise, you should hire a knowledgeable hardware consultants and employ them on a regular basis, since your needs will probably change as time goes by and your hardware will likewise be forced to adapt as well.