Separate PHP Error Logs for Multiple Domains with Plesk

I've put a few additional domains on my Media Temple (dv), and it's been great. One thing that was bugging me was not having an easy way to tell which domains were generating which scripting errors. When it was just my site, things were easy to follow, but with the addition of a BBS and a couple other WordPress installations, it was getting difficult to tell who was complaining about what.

Solution

You can configure PHP to have a separate error log file for each VirtualHost definition. The trick is knowing exactly how to set it up, because you can't touch the configuration directly without breaking Plesk.

Every domain name on your (dv) has its own directory in /var/www/vhosts. A typical directory has the following top level directories:

cgi-bin/
conf/
error_docs/
httpdocs/
httpsdocs/
...and so on

You'll want to create a vhost.conf file in the domain directory's conf/ folder with the following lines:

php_value error_log /path/to/error_log
php_flag display_errors off
php_value error_reporting 6143
php_flag log_errors on

Change the first value to match your actual installation. After you're done editing the vhost.conf file, test the configuration from the console with:

apachectl configtest

And finally tell Plesk that you've made this change.

/usr/local/psa/admin/bin/websrvmng -a

Web-Based PHP Error Log Browsing

On my old host, FutureQuest, they had a nice built-in PHP Error Log viewer in the control panel. I thought it would be neat to make something similar, so I created a portable PHP script to help me monitor each domain separately.

To try it out, first download the script, unzip it, and then copy the entire folder to some place on your webserver. Then, point your browser to the folder and follow the directions. You can also read the README.TXT file for directions. The script does some tests to make sure things are set up correctly, going so far as to insist that you password protect your directory. You can disable this check in the script directly, but you should probably do this to at least keep casual visitors from nosing around the sensitive parts of your server.

» Download error_log.zip

Related Links

• How to configure PHP for error logging
• How to modify the Plesk vhost.conf file so everything plays nice.

Other Optimizations for the Media Temple (dv) Base

Some additional memory usage reductions I've made, after applying these configuration changes to my VPS at MediaTemple.

Limiting number of Plesk PSA connections

When you go to your Plesk administration page, you're accessing a special webserver over SSL. The webserver process is httpsd, and it's larger than the stock httpd process, using up another precious 30MB.

To reduce its memory usage, I've played around with reducing the number of clients it spawns and keeps around to 1. It never actually seems to drop below 2, but at least it doesn't balloon up.

The Plesk-safe way to do this is to create a file /usr/local/psa/admin/conf/httpsd.custom.include and add the following overrides:

MinSpareServers 1
MaxSpareServers 1
StartServers 1
MaxClients 1
KeepAlive On
KeepAliveTimeout 10

This reduces the number of services that are running to a minimum. Note that if you have other people running Plesk control panels, you may run into login problems and sluggish behavior if you reduce the # of servers by too much. In general it's just me on my server, so I keep the numbers low.

Check that everything is ok with:

/usr/local/psa/admin/bin/httpsdctl configtest

Type free to see how much used memory there is before you start, then restart the Plesk webserver (gracefully) with:

/usr/local/psa/admin/bin/httpsdctl graceful

Type free again to see if you're using less memory.

Disable Spam Assassin

SpamAssassin is a service that filters for spam on incoming mail, and each instance uses 21MB. I'm using other spam-prevention methods, so I've turned it off.

• To disable it in Plesk: Go to Server:Services Management and click the red STOP icon.
• From the shell: Type /etc/init.d/spamassassin stop from the shell.

To disable it permanently, remove execution permission from the spamassassin script by typing chmod a-x /etc/init.d/spamassassin...you can always add it back by change the - to a + in the chmod command example. Note: I haven't tested this recently by restarting the server with the shutdown -r command...I think it works.

Disable DNS Services

If you're not using DNS on your (dv), then you might as well turn it off as it uses a smidgeon of memory.

• To disable it in Plesk: Go to Virtuozzo:System Services and click the named hyperlink. Click "stop service** and "disable autostart".
• From the shell: Type /etc/init.d/named stop. You can also remove the execution permission from the named script to prevent it from starting on a server reboot.

In case you're wondering if this will take your server down, note that Media Temple already provides DNS for your server, through the Media Temple AccountCenter. The DNS server settings in Plesk are for specialized applications that go beyond simple hosting. Your subdomains will not stop working.

UPDATES

The good folks at Media Temple have researched these tweaks themselves, and have released an official unsupported advanced method of doing it the right way! Awesome! :-)

Monitoring My Media Temple (dv) Base Memory Usage

Screenshots

I'm still figuring out exactly what all these parameters tell me, so take this section with a grain of salt.

This is the output of cat /proc/user_beancounters.

• The two blue lines, privvmpages and physpages, are the amount of current allocated memory and actual used memory respectively. The
• The two orange lines vmguarpages and oomguarpages tell you how much memory can be allocated ("guaranteed available").
• The purple line shows othersockbuff, which is memory used for interprocess communication between (among other things) mysqld and other programs (like Wordpress). I was getting a lot of failcnt here because of a bad table in Wordpress as I mentioned above.

The columns:

• held/maxheld are the current amounts of memory (in bytes) or pages (in 4K chunks) are being used at the time that /proc/user_beancounters was dumped. It changes from moment to moment. Maxheld, I think, is the "highest" amount requested in recent history.

• barrier is the maximum amount of that resource that your VPS instance can request, unless there is additional shared resources available. The limit is the absolute maximum that can be requested.

Now let's look at the top command output.

The highlighted green areas show memory and CPU information. The mem total line shows the total available, which corresponds to privvmpages. The mem used line corresponds to physpages.

If you look in the display down a bit, you'll see VIRT and RES, which shows you how much memory is being used by each running process, also in terms of allocation (VIRT) and actual use (RES). This isn't an exact count, though; if you sum up the numbers you'll get more than mem total and mem used. That's because a lot of code is shared and used by more than one process.

One thing you'll want to look for is to make sure you're not seeing too many running processes competing for memory. For example, count up the number of httpd (web server) processes and smtpd (mail) processes. You'll also want to see which processes might be hogging the CPU by looking at the cpu column; as top updates, the most hoggy processes will be shown at the top by default unless you have changed the sort order, which I never remember how to do. If you see something taking 100% CPU for an extended period of time, that's a good sign something isn't right.

So my VPS is supposed to have 256MB of memory allocated to it. So WHY THEN does mem total exceed that and say I have 689MB? That's because mem total includes swap disk space, though the swap disk doesn't show up in the VPS directly. To ensure that I don't go to swap, which is slower, I make sure that mem used doesn't exceed 256MB. The whole beginning of this article describes what I did to balance the resources.

Finally, the load average at the upper left (in orange) shows some nice low numbers. A range between 0 and 1 is pretty lightly loaded, 1-3 is moderately loaded, and above 3 you're starting to see some strain.

Don't like using the shell? Here's what it looks like in Plesk 8.1:

Here's what mine looks like when I click Server (left panel) then Statistics (first line, under System). This is just the first line:

You should recognize the mem total and mem used values here, which also correspond to privvmpages and physpages. Remember that a "page" is 4K.

I actually haven't seen the Shared and Buffer values go up before...they may have before I constrained my server configuration. I would rather stay within my allocation, though, and try to achieve a level of consistent performance.

There's no swap disk in use here. It's a virtual server, so there's no actual swap disk. The Virtuozzo system takes care of this behind the scenes (I think).

Now, if you go to the Virtuozzo panel, then Resources -> Extended, you'll see something like this below:

These correspond to the userbeancounters I was talking about in the first picture. It's a little friendlier to look at that the raw text output I showed earlier. soft limit is the same as barrier, and hard limit is the same as limit.

Monitoring Server Resource Usage

One thing to remember about these values is that they're constantly changing. Finding out what's causing a particular memory overage means that you have to watch how these values change in real time. I did this by opening a couple of shell windows, then running top and cat /proc/user_beancounters in them. There may be a more sophisticated way of doing this, but I'm a noob at this stuff.

There are a few logfiles I monitored simultaneously in other shell windows. Here's what my setup looked like while I was watching:

The top left window shows filtered output from the apache log, which I do with the following command line: tail -f ~user/statistics/logs/access_log | grep -o -f ~/match-web

~user is the "ftp user" defined in Plesk for your particular domain, and ~/match-web is a file in the root user's directory that contains the following lines:

"http:.*/"
GET.*HTTP

This makes the live output from the web server a little more focused on pageloads, so you can get an idea how much is going on.

The top right window shows output from the current system message log, with the following command: tail -f -n50 /var/log/messages

On my server, the message log rotates between messages.1, messages.2, etc, so you might have to dig for it. Anyway, watching this log in real time shows me what's going on, like smtp connections (email) and attempts to break into the server by automated scripts. It's very irritating to watch, but it allowed me to figure out that it was smtp that was creating some problems in conjunction with the other windows.

The two bottom left windows are the top command and while true ; do cat /proc/user_beancounters ; sleep 1 ; done command. If I saw a sudden surge in %CPU or an increase in the failcnt (in this screenshot you can see that the othersockbuf problem was still happening), I could look at the top two windows and see what happened.

Finally, the bottom right-window is my Mint installation, showing me the traffic stats for my website. This gave me an idea of how heavily loaded the server was, and I could monitor the various windows to see what was going on.

Other Reading

in progress

• OpenVZ
• UBC Parameter Descriptions
• Linux Memory Usage
• UBC Explained
• How to Profile Memory in a Linux System
• Use the 'pmap' (process map) command to see the memory breakdown of a specific process. Or look in '/proc/pid/status ' and '/proc/pid/maps'.
• Use the 'pstree' command to show processes in a tree hierarchy, showing who spawned what
• Troubleshooting Linux Memory - similar in some ways to the optimization articles I've written.
• To see what sockets are being held by what apps, use lsof (list open files) or netstat -p. An article on the use of lsof describes it in the intrusion detection role, but it's still useful. Here's another lsof quickstart. And this one describes how to recover deleted files using lsof (at the end).

Configuring a Media Temple (dv) Base for WordPress

This article is still being written, but please feel free to comment on it if you've stumbled upon it

I'm running a (dv) Base system from Media Temple, which is their 256MB version. This is their lowest-end server configuration, which is actually a virtual server sharing the same hardware.

As my blog has grown in popularity, when I switched over I got enough traffic to overwhelm the server. It turned out that the base configuration isn't tuned to work within the guaranteed 256MB of memory allocated. It's configured to run on a considerably larger server, and grabs extra memory as needed if it's available from the global memory pool. This is all fine and dandy when memory is available, but if it ISN'T, the server will start to chug.

After making the following tweaks to my server setup, one of my articles was dugg and the server was hit by 2750 pageloads per hour for a few hours. This is a pretty significant amount of traffic, and the server had no problems staying up. In fact, it barely broke a sweat. Keep in mind though that in addition to the server tweaks below, I also had optimized my wordpress installation with WP-Cache, which reduces server load significantly. You can read more about this in WordPress and Shared Hosting.

Anyway, here's what I've done to get my DV in line. I'll write about the WordPress issues a little later. If this article convinces ya to get a (dv), refer my site when you order so I get a little reward from Media Temple :-)

Check for Memory Faults

The virtual server is a product called "Virtuozzo", and there's a special file that tells you how much memory you're allowed to use. When you exceed that amount of memory, a fault is generated. Ideally, you don't want any faults.

To check up on this, you can use Plesk (under Virtuozzo) or login to your server and type cat /proc/user_beancounters into the command shell. The column that's of most immediate use to you is failcnt, which tells you how many times a program has tried to get some memory to do something, but wasn't able to do so. The values should all be 0.

I'll fill this section in a little later, because it's kind of tricky to understand and I don't entirely get it. Here's a cheesy script I use to constantly monitor the user_beancounters to see if they're going up (I just open a second shell window):

#!/bin/sh
while true ; do cat /proc/user_beancounters ; sleep 1 ; done

When you run this script, it just cats the output every second (see image). If you're monitoring other parts of the server, you might see some overall patterns in what seems to be causing the memory usage. There's some more information at the very end of this post (with screenshots) to help orient you.

Apache Configuration

The default configuration for Apache allowed too many instances of httpd to spawn. Since each instance takes between 20MB and 40MB, you don't want more than your available memory to be created. If I figure on allowing 128MB available out of my 256MB, and each Apache instance takes 20MB, that means I can't have too many of them active at once. In actuality, the memory usage reported by ps aux under the VSZ and RSS headings isn't the real picture of memory usage (see this article for some insight into this), so I can actually have more instances. I experimentally determined that 20 seemed to be enough without pushing the server too hard.

The pertinent settings are in the prefork section of /etc/httpd/conf/httpd.conf. Make sure you save a backup of the file. Here's what my prefork section looks like (SAVE A BACKUP BEFORE YOU EDIT!)

<IfModule prefork.c>
StartServers 1
MinSpareServers 1
MaxSpareServers 3
ServerLimit 20
MaxClients 20
MaxRequestsPerChild  4000
</IfModule>

UPDATE: On the week of August 21, 2007, I noticed that my memory allocation was exceeded regularly by about 100MB or so. It seems that the way (MT) is calculating physical memory use is also taking virtual memory into account. I had to drop ServerLimit from 20 to 6 to maintain responsiveness. I have a ticket open to see if this is a bug or not (virtual memory is disk-based, so you'd think it wouldn't count against your physical memory allocation). :END UPDATE

UPDATE:Chris McKee points out that Media Temple has a performance tuning knowledge base article with some recommendations for the Apache configuration. My settings are much more conservative, designed to avoid memory swapping altogether.

I also disabled modules that I wasn't using by commenting them out. I reduced the memory footprint of each httpd instance by a bit. Every bit helps! The lines with the # in front are disabled modules.

LoadModule access_module modules/mod_access.so
LoadModule auth_module modules/mod_auth.so
# LoadModule auth_anon_module modules/mod_auth_anon.so
# LoadModule auth_dbm_module modules/mod_auth_dbm.so
# LoadModule auth_digest_module modules/mod_auth_digest.so
# LoadModule ldap_module modules/mod_ldap.so
# LoadModule auth_ldap_module modules/mod_auth_ldap.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
# LoadModule mime_magic_module modules/mod_mime_magic.so
# LoadModule cern_meta_module modules/mod_cern_meta.so
LoadModule expires_module modules/mod_expires.so
# LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
# LoadModule usertrack_module modules/mod_usertrack.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
# LoadModule dav_module modules/mod_dav.so
# LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
# LoadModule asis_module modules/mod_asis.so
# LoadModule info_module modules/mod_info.so
# LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
# LoadModule imap_module modules/mod_imap.so
LoadModule actions_module modules/mod_actions.so
# LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
# LoadModule proxy_module modules/mod_proxy.so
# LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
# LoadModule proxy_http_module modules/mod_proxy_http.so
# LoadModule proxy_connect_module modules/mod_proxy_connect.so
# LoadModule cache_module modules/mod_cache.so
LoadModule suexec_module modules/mod_suexec.so
# LoadModule disk_cache_module modules/mod_disk_cache.so
# LoadModule file_cache_module modules/mod_file_cache.so
# LoadModule mem_cache_module modules/mod_mem_cache.so
# LoadModule cgi_module modules/mod_cgi.so
LoadModule logio_module /usr/lib/httpd/modules/mod_logio.so

After you make these changes, type apachectl configtest to make sure you haven't screwed something up. If you get the Syntax OK go-ahead, then apachectl restart will restart the server and reread the configuration.

Note: Your website might actually use some of the modules that I'm not using, so be careful.

Note: ps -A -o pid,vsz,rss,pmem,time,comm shows memory usage (VSZ and RSS) roughly, though it's not really accurate.

SMTP connections

SMTP is used to receive mail addressed to mailboxes on your server. It's also used to send mail to other people from your email account.

By default, the number of incoming SMTP connections was set to "unlimited". When I was getting hammered by spam, this created a LOT of simultaneous connections, each of which took up memory. I limited the number of instances to 10 by editing /etc/xinetd.d/smtp_psa, from the default of 0 (unlimited).

I also noticed that when a secure SMTP connection (SMTPS) occured, the SMTPS daemon would use 100% CPU. I don't use secure SMTP, so I denied all SMTPS connections to prevent spammers connecting to the secure port from creating this problem. That file is /etc/xinetd.d/smtps_psa.

After you make these changes, type /etc/init.d/xinetd restart to tell the server to re-read and use the new configuration information.

POP3 and IMAP SSL

Since I don't use SSL for IMAP or POP (these are services that email programs use to download your mail), I disabled these scripts as well by commenting out the lines that start with report_action "Starting imap-ssl" and report_action "Starting pop3-ssl". I will have to look back at these later to see if they really make a difference.

To restart the services, type /etc/init.d/courier-imap restart.

MySQL Configuration

Although I wasn't sure if MySQL was a problem, I decided to configure it to run in a smaller memory footprint. I'm not an expert in MySQL configuration, so I just used the "medium" memory configuration as a starting point. You can find the my-medium.cnf file, suitable for a 64MB instance, in /usr/share/doc/mysql-server-4.1.20/. The changes I made were in the [mysqld] section of the file /etc/my.cnf

[mysqld]
set-variable=local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1
skip-bdb

set-variable = max_allowed_packet=1M
set-variable = key_buffer=16M
set-variable = table_cache=64
set-variable = sort_buffer=512K
set-variable = net_buffer_length=8K

set-variable = myisam_sort_buffer_size=8M

I basically added the set-variable commands.

When I was importing my WordPress database, I also had to increase max_allowed_packet to 16MB. It turned out that there was an inefficiency in my WordPress setup that actually was the source of this problem: The Tan Tan Reports plugin created a very large entry in the wp-options database table. I removed the plugin and its data from the wordpress database table and I was able to reduce it back to 1MB. This was very important from a memory usage perspective too; MySQL allocates extra memory for inter-process communication (like, between MySQL and WordPress), and this caused memory failures in the user_beancounters!

After you make these changes, type /etc/init.d/mysqld restart to reload the MySQL configuration.

Additional Optimizations

There are some additional things I did, like disabling spamassassin (21MB), limiting the number of plesk connections (another 30MB), and disabling named (1MB). You can read about these additional optimizations over here.

Screenshots

See this page for some screens and descriptions of how I am monitoring memory usage the old-fashioned way.

WordPress and Shared Hosting

After I getting linked by several high-profile websites, the increased traffic has forced me to move twice. These notes describe my own experiences with WordPress and its needs. Individual WordPress installations vary in complexity, so your mileage may vary.

An Introduction to Blog Hosting Requirements

I was originally on Pair Networks, a reliable host that I've been with for almost 10 years. My website presence was largely for personal and business use, handling email only. This changed in late 2004, when I started my blog Better Living through New Media. Readership was up to about 100 people a day, through organic search and friends visiting, and Pair's inexpensive Advanced web hosting plan was quite adequate. At $17.95/month, it is considerable more expensive than other popular WordPress hosts like DreamHost, BlueHost, A Small Orange and TextDrive, but it was the host I've been with for years. The WordPress developers have a list of webhosts also that you may want to look at.

When you are starting out, your needs will be pretty basic:

• Disk space, measured in megabytes (MB) or gigabytes (GB, which is 1000MB). The WordPress install only takes 3MB, and custom themes may add another megabyte. If you're hosting lots of photos, podcasts, or videos, you'll want more space, but just about any hosting plan will give you enough space. My entire WordPress install is about 500MB currently.

• Bandwidth, measured in GB/month or TB/month (terabytes, which is 1000GB). For a site getting started, you'll probably see less than 1GB a month of "normal" use. However, it's the abnormal conditions that you'll have to look out for.

• Apache, MySQL, and PHP are the base requirements to run WordPress on a web server. In the case of Pair, the cheapest plan that includes PHP and MySQL is the plan I was using.

The focus of this note is not installing WordPress, but on the traffic levels and server load issues that may drive you to move. For installation help, visit the WordPress Codex.

Getting Started: Low Levels of Traffic

As I've said, you can get started with just about anything. When I started blogging, I installed StatCounter and used this to monitor the amount of traffic I was getting. One nice thing about StatCounter is that is also gives you a nice way of exploring the patterns within the traffic, which gives you an idea of what people are looking at. The three fundamental stats of web traffic are pageviews, unique visitors, and referrals. A pageview is a page loaded with all the accompanying graphics and text. A unique visitor is a person who's been looking at your pages. A referral is where the person "came from", for example another website or perhaps a search engine.

While these stats are all interesting, the one that matters when it comes to choosing a hosting plan is pageviews. When I got started, I had a count of about 10-20 pages a day, which rose to around 100-200 pageviews over the next year. The site remained responsive and snappy...though the pageviews were a lot to me, the server handled this without any problem.

First Sign of Strain: 500 Pageviews A Day

The first sign of trouble was when I hit the 500 pageview-a-day mark, when I noticed the server becoming noticeably slower in delivering webpages. I attributed this to WordPress, which is a dynamic web application. Unlike a regular HTML page, WordPress pages are created "from scratch" every time someone requests one by pulling data out of a database and formatting it on-the-fly. This can become very slow, especially if you've added plugins like Markdown that reprocess all your text.

To deal with the problem, I had two choices: move to a faster server and pay more money, or address the inefficiencies of the WordPress page model. I chose the latter approach, installing the WP-Cache plugin that essentially stores copies of your pages so WordPress doesn't have to create them from scratch. This reduces the strain by a significant factor, and allowed me to stay with Pair for another year.

Time to Move: 1500-2000 Pageviews a Day

I started experiencing more serious problems when I hit the 1500-2000 average pageviews-a-day mark, particularly during peak hours. This is the time of day when the most people are visiting your website at the same time. Your server has to work its hardest, handling many simultaneous requests for web pages with what it has. It's just like rush hour: too many cars on limited road space results in slower traffic flow for everyone. With shared web hosting, there are also other websites sharing your machine, so peak effects can be multiplied.

I started noticing that webpages were taking a long time to come up, and that logging into my admin page to write posts sometimes failed. I also started seeing database connection errors, which indicated that I had exceeded my allotment of bandwidth to the database server. While looking up information about the Pair database servers, I saw that the general consensus was that they are slow compared to other hosting companies. Also, the use of the Mint real-time web traffic statistics package, which had replaced StatCounter for my daily use, was expressly forbidden by Pair's terms of service. They hadn't said anything about it, but the additional database strain added by Mint was probably taking away from overall site performance. It was clearly time to move.

Host Evaluation: Pair shared hosting is great if you're serving lots of static web pages or a small number of dynamic pages through PHP/MySQL.

Picking the New Host

Pair is one of the best web hosts out there, in my experience very responsive and reliable. One caveat: I'm more technical than the average person when it comes to this kind of stuff, familiar enough with server administration that I don't need to ask for a lot of help. I wanted my new hosting provider to have comparable quality of service at an affordable price.

As I looked around, I was struck by many of the complaints and horror stories that were going around practically every host I've mentioned. The only two exceptions were Pair (almost nothing bad said about it, but then again it caters to a more technical crowd) and FutureQuest, which was described as having even better customer support with faster database servers. I did another gut check, which was to google for the phrase "Futurequest sucks". If you try the same thing with other popular hosts, you'll usually see a lot of complaining. Not the case with FutureQuest, so that's who I went with. I moved my files over, restored my database, and I was back in business.

The main issues with FutureQuest were much lower bandwidth and disk space allocations. At the time I moved my main website from Pair, I was on their Webmaster plan for $29.95/month. Bandwidth allocation was 200GB/month, and disk space was 3000MB. My new plan at FutureQuest was $26.95 a month, and included only 20GB of transfer bandwidth and 700MB of disk. At the time, my website was averaging about 15GB of transfer (checkable through the monthly web logs), so I figured I was OK.

The months I spent with FutureQuest were very good. Although I had a few issues with the older version of MySQL they use, once that was cleared up I found the website back to its snappy self. The customer service was everything I had heard, incredibly responsive and proactive. The account control panel, which allows you to do things like set up email accounts and check the status of your service, was also superior to what Pair has in place. I was serving 2500 pageviews a day on average without noticing any degradation in service.

Handling Big Traffic Spikes

The first sign of trouble struck in mid December 2006, when I released a new version of my yearly calendar. It was picked up by the popular site LifeHacker, which immediately drove thousands of people to a particular page on my website during peak hours. The proactive technical staff at FutureQuest noticed immediately that the server was being strained, and promptly emailed me a notification. After 15 minutes, the server was about to completely meltdown, so they disabled my account to keep the websites from the other customers from being dragged down. They emailed me a second time to get back to them as soon as possible.

One problem with having your account disabled is that you have no way of accessing your website or your email. Fortunately I keep my computers running instead of turning them off, so I had a few email reports about incoming traffic that allowed me to see where the unexpected traffic was coming from. I told FutureQuest this, and described the course of action I would take:

• Upload a static version of the webpage, so WordPress wouldn't have to deal with it. Most people following a link will just visit the first page.
• Move the downloadable files associated with the webpage to my old Pair account, which I had maintained because I host other small personal websites there for friends and family.

FutureQuest re-enabled the site within minutes of receiving my email, and I stepped in and made the changes immediately. This allowed the website to stay up with acceptable levels of server load, which is measured in reference to the number 1. Anything less than 1 is a "lightly loaded" server; you can see this number by typing uptime if you're logged into the server with a terminal program. Anything greater than 1 and 2 is "moderately loaded", and if you start seeing numbers greater than that you can expect some sluggishness. In the case of my website, it was causing load spikes of 20-100. After applying the changes, it went back down to a more acceptable 4 to 6: still bad, but not catastrophic. I saw my highest traffic spike ever for the day: 25,000 pageloads.

Typically, the first two days of a spike like this are the highest, and after about a week traffic settles back to normal. The days afterwards you will see elevated levels of traffic from all the secondary linking that occurs from other websites and blogs. All that linking does not go unnoticed by search engines, which raises your site profile in their ranking algorithms and drives increased traffic through search. Because of this extra burst of popularity, I was in danger of exceeding my bandwidth allocation by 5GB. FutureQuest sent me an email based on their predictions, and I upgraded to the next higher-level plan for $35.95/mo. This plan's traffic allocation was 30GB, 10GB more than my old plan.

After all this excitement I thought things would settle back down, but I got linked again a week later, this time to a different part of my website. This time, FutureQuest told me that I needed to either upgrade to a dedicated server ($200 setup, $200/month) or move to avoid these continuing "chronic" problems caused by my website. They also warned me on a separate occasion that my projected bandwidth use was going to be more than double my plan, which would cost me an additional $150 in overage charges if that happened.

I still am very impressed by how their customer service, though in my case it was bad news, was handled so well. I completely understand that being on a shared host with other people entails a certain accountability; I was using far more than my fair share of the server and compromising the level of service for everyone. It was time to move again...$200/month is far too much for me to afford. And $35.95/month for a mere 30GB of transfer is way too low. Compare that to the 100-1000GB allocations you get with other hosting.

Host Evaluation: FutureQuest has phenomenal customer service and technical monitoring for a shared host. However, it's expensive. If you have a smaller website or can afford to pay, it's pretty great. I was sad to leave.

Aside: Website Efficiency

When you get a lot of traffic in a day (and 25K isn't even that high) you really stress the available resources in your service. I had gotten some large spikes before, but nothing on this level. It exposed several inefficiencies in my setup:

• The Mint stats package I was using was the source of the server spike. A third-party plugin I had installed was doing some simple download counting in a very inefficient manner, which caused memory use to skyrocket. Removing this plugin brought things down to an acceptable level.

• Secondly, increased levels of comment spam were causing high loads on the server as well. I was getting well over a thousand comment spams a day, sometimes as many as 3000, in a single day. Those 3000 spams each have to be filtered by the anti-spam plugins I have running, which themselves burn up server resources. The stats package I'm using does not count these as pageviews, but they do create spikes of use that are problematic.

As a website grows and becomes more feature-laden, your traffic and server load problems increase. If you don't have the time or inclination to learn how to deal with it, it's going to cost money. Either you buy a big dedicated or managed server package ($100-$300/month) or hire someone to handle your server administration for you.

It's important to note that most hosts are not as proactive as FutureQuest. The 15-minute email turnaround time for FutureQuest in handling my website downtime is incredible; you'd have to wait 24 hours to 3 days before getting someone to work with you to resolve your problem. And that bandwidth usage warning email? Most hosts will charge you AFTER you've incurred the overage, which you had no idea was happening, and then you're screwed.

Moderate Levels of Traffic: 3000-5000 pageviews a day

I'm still experiencing increased levels of traffic from those recent surges, and am averaging around 3000-5000 pageviews a day. This is considered by people I've talked to as "significant" website traffic. I was thinking it was time to get away from shared hosting.

When you hit 3000-5000 pageviews a day, you're also at the point where shared hosting isn't enough anymore. Your only option, short of finding a buddy to give you free server space, is to buy dedicated hosting. You get an entire server devoted to your site, but since you're not longer sharing the machine with other people you have to foot the entire bill. Typically, it's between $100-$200 a month, and you're also responsible for keeping it running. There is an even more expensive kind of dedicated hosting called managed hosting, where not only will you get the entire machine to yourself, they'll also run it for you and guarantee its operation.

Since I don't have the hundred bucks a month to spend, I went with Media Temple's dedicated virtual hosting for $50/month. Media Temple is a mainline hosting provider, and provides much more generous disk and bandwidth allocations. MT also offers another hosting plan called the Grid-Server, which takes shared hosting to another level by sharing hundreds of servers with hundreds of websites. Normally, a shared server has a number of customers assigned to it, with each server's resources distinct from each other. With the grid-server technology, the granularity is finer, and the resources from all servers can be put into one common pool. Theoretically, this means you can get fantastic power and value for one low cost. However, this is a new technology and MT is still working out the bugs. I decided to stay with dedicated virtual hosting.

Dedicated virtual hosting is like getting an entire machine to yourself, but in actuality it's a form of shared hosting. The server is actually a software "container" that looks like a stand-alone server, sort of like when a large house is divided into several condominiums. You are guaranteed a certain amount of memory, and if there's any left over you can "burst" to use what's available. Another advantage of dedicated virtual hosting is that you can easily backup your entire server (it's virtual) and you can configure anything you want on it. That's also the downside, because you're also responsible for keeping the thing running. In general this isn't as bad as it sounds, as you can administrate your server using a web-based control panel. However, since I'm using the very cheapest level of dedicated virtual server, I find that there's still some things I need to do to ensure it runs well.

Media Temple sells three levels of dedicated virtual servers, the (dv) Base (which is what I'm using), the (dv) Rage, and the (dv) Extreme. Each server doubles the resources available from the previous level, so the (dv) Rage has twice the memory and disk storage of the (dv) Base. The Base has 256MB of memory, which I know from experience is plenty to run a small server that doesn't have to use graphics. What I didn't know was whether this server would be capable of handling my 3000-5000 pageview-a-day habit.

After a few days of struggling with it, I have it at the point where I'm pretty confident that it'll handle 5000-10,000 pageviews a day without problem. The problemw as this: while the (dv) is set up and ready to go the moment you order it, my particular installation was configured to use as much memory as it could. As I mentioned before, the Base has 256MB of memory, but this is "guaranteed" memory. The virtual server system that is managing all these (dv) virtual servers has a LOT more memory than 256MB, and this is a pool that everyone can draw from. That means if my server needs more than 256MB, and the memory is available, it can get it. If it's NOT, then the memory request fails. The ideal case, from my perspective as someone who wants my server to run well all the time, is to have it all work within my guaranteed memory allocation. From MT's perspective, if the server is telling me I am running out of memory, I am more likely to upgrade to a more expensive server package.

My solution was to spend several days going blind learning how to do basic linux server administration. I configured the mysql database engine to work within 128MB, and limited the number of simultaneous web connections to something reasonable while trimming the number of "modules" loaded by the web server. I also discovered several inefficiencies in the way my WordPress installation was set up relating to poorly-written plugins. I've been monitoring the server live for a few days now, checking the load average as page requests roll in, but these are all subjects for another article.

Another advantage of the (dv) is that you can resell web service. The number of domains you can add to your site, for example, is up to you. You can add up to 30 before you need to buy an additional "Plesk license"; theoretically, I think you can work around this by manually configuring the server. It normally costs money to add new domains to your web account with shared hosting. With 20GB of space, you can afford to bring a few friends on and offset the cost of your hosting, if you're willing to do the administration. The Plesk control panel, which allows you to adminstrate the various functions of your server, can also be configured to show your clients whatever features you think they need. It's a very nice system, and I'll write up my impressions at a later date.

Host Evaluation: Media Temple offers a great value, and they're quite well known in the community. I happen to also get a discount on the service through the 9rules affiliate program, so this makes the service comparable to what I was paying for FutureQuest.

The Next Level

I haven't yet been able to test my server under higher loads. I would guess that's the 10K pageviews-a-day level. If my (dv) holds up to it, then I'm set for a while.

What might change the equation is if I start hosting other websites or add some web applications to the mix. As it is now, I'm running just a single application---WordPress---on this server. If I added a busy BBS or commerce site, then I'll be pushing the limits of the server again, and will be forced to upgrade. At that point, though, I would hope to be making enough money from the Web to justify the cost of a real dedicated server.

Here's how my pageviews vs hosting worked out. Your numbers may be different, based on the number of downloads and images you're hosting on your website.

So that's my thoughts on the subject. This page will be updated as I learn more. Enjoy!