Handling Big Traffic Spikes
The first sign of trouble struck in mid December 2006, when I released a new version of my yearly calendar. It was picked up by the popular site LifeHacker, which immediately drove thousands of people to a particular page on my website during peak hours. The proactive technical staff at FutureQuest noticed immediately that the server was being strained, and promptly emailed me a notification. After 15 minutes, the server was about to completely meltdown, so they disabled my account to keep the websites from the other customers from being dragged down. They emailed me a second time to get back to them as soon as possible.
One problem with having your account disabled is that you have no way of accessing your website or your email. Fortunately I keep my computers running instead of turning them off, so I had a few email reports about incoming traffic that allowed me to see where the unexpected traffic was coming from. I told FutureQuest this, and described the course of action I would take:
- Upload a static version of the webpage, so WordPress wouldn’t have to deal with it. Most people following a link will just visit the first page.
- Move the downloadable files associated with the webpage to my old Pair account, which I had maintained because I host other small personal websites there for friends and family.
FutureQuest re-enabled the site within minutes of receiving my email, and I stepped in and made the changes immediately. This allowed the website to stay up with acceptable levels of server load, which is measured in reference to the number 1. Anything less than 1 is a “lightly loaded” server; you can see this number by typing uptime if you’re logged into the server with a terminal program. Anything greater than 1 and 2 is “moderately loaded”, and if you start seeing numbers greater than that you can expect some sluggishness. In the case of my website, it was causing load spikes of 20-100. After applying the changes, it went back down to a more acceptable 4 to 6: still bad, but not catastrophic. I saw my highest traffic spike ever for the day: 25,000 pageloads.
Typically, the first two days of a spike like this are the highest, and after about a week traffic settles back to normal. The days afterwards you will see elevated levels of traffic from all the secondary linking that occurs from other websites and blogs. All that linking does not go unnoticed by search engines, which raises your site profile in their ranking algorithms and drives increased traffic through search. Because of this extra burst of popularity, I was in danger of exceeding my bandwidth allocation by 5GB. FutureQuest sent me an email based on their predictions, and I upgraded to the next higher-level plan for $35.95/mo. This plan’s traffic allocation was 30GB, 10GB more than my old plan.
After all this excitement I thought things would settle back down, but I got linked again a week later, this time to a different part of my website. This time, FutureQuest told me that I needed to either upgrade to a dedicated server ($200 setup, $200/month) or move to avoid these continuing “chronic” problems caused by my website. They also warned me on a separate occasion that my projected bandwidth use was going to be more than double my plan, which would cost me an additional $150 in overage charges if that happened.
I still am very impressed by how their customer service, though in my case it was bad news, was handled so well. I completely understand that being on a shared host with other people entails a certain accountability; I was using far more than my fair share of the server and compromising the level of service for everyone. It was time to move again…$200/month is far too much for me to afford. And $35.95/month for a mere 30GB of transfer is way too low. Compare that to the 100-1000GB allocations you get with other hosting.
Host Evaluation: FutureQuest has phenomenal customer service and technical monitoring for a shared host. However, it’s expensive. If you have a smaller website or can afford to pay, it’s pretty great. I was sad to leave.
Aside: Website Efficiency
When you get a lot of traffic in a day (and 25K isn’t even that high) you really stress the available resources in your service. I had gotten some large spikes before, but nothing on this level. It exposed several inefficiencies in my setup:
- The Mint stats package I was using was the source of the server spike. A third-party plugin I had installed was doing some simple download counting in a very inefficient manner, which caused memory use to skyrocket. Removing this plugin brought things down to an acceptable level.
Secondly, increased levels of comment spam were causing high loads on the server as well. I was getting well over a thousand comment spams a day, sometimes as many as 3000, in a single day. Those 3000 spams each have to be filtered by the anti-spam plugins I have running, which themselves burn up server resources. The stats package I’m using does not count these as pageviews, but they do create spikes of use that are problematic.
<
p>As a website grows and becomes more feature-laden, your traffic and server load problems increase. If you don’t have the time or inclination to learn how to deal with it, it’s going to cost money. Either you buy a big dedicated or managed server package ($100-$300/month) or hire someone to handle your server administration for you.
It’s important to note that most hosts are not as proactive as FutureQuest. The 15-minute email turnaround time for FutureQuest in handling my website downtime is incredible; you’d have to wait 24 hours to 3 days before getting someone to work with you to resolve your problem. And that bandwidth usage warning email? Most hosts will charge you AFTER you’ve incurred the overage, which you had no idea was happening, and then you’re screwed.
9 Comments
David,
thanks for writing this up. Could you expand on what plug-ins were giving you performance trouble (aside from Mint) and what you did (turn off the plugin, upgrade it, use another to do the function)? Also, since I’m thinking of moving to MT’s (gs) service, any link to issues with it would be very helpful.
Thanks for this excellent post. I can definitely sympathize with your plight, as I recently moved from shared hosting to a managed VPS.
I would be interested in learning more about the resources you used to enhance your linux skills (books, websites, articles, etc.).
Rick: It was the DLoads plugin in mint that was the main culprit. Another one was Tan Tan Reports, which creates a very large entry in the wp-options table, causing mysqld to hiccup. As for the (gs), I don’t know about it…I avoided it because I keep seeing issues about it, plus I wanted to try the (dv) because the price dropped so much. Try searching Technorati for it, and I’m sure you’ll find a lot of commentary on it.
Richard: When I first got started, I had a book called the Unix Systems Administration Guide and a very “traditional” Linux distribution (“distro”) called Slackware. I spent a day going through every file in the
/etcdirectory understanding what it was, looking up what I didn’t know. Thank goodness for the Internet. This was back in 1998. So I learned basic concepts that way, and gave me a framework for understanding.The challenge is that every linux distro is different in where it puts its configuration files. I’ve started compiling a list of the ones for the Plesk/CentOS setup that the (dv) uses, which I’ll post when I get some time to write it up.
What are you using for stats now? BTW – I’m just getting with WP so this info is invaluable.
I’ve re-introduced Mint, but just with the SessionTracker and Outclicks peppers. I’m also using Google Analytics and StatCounter, both online services that are free. I’ve heard interesting things about HitTail too, but haven’t tried it yet. You can read about my experience with this in my post Mint vs Google Analytics (and StatCounter).
Dave,
Thanks for the recommendations. I’m looking forward to the configuration file summary, as my host is also running CentOS.
Have you had any difficulties working with Plesk? My VPS host gave me the option of Plesk or CPanel. I originally chose Plesk, but was encountering many issues with PHP open_basedir, includes and file permission issues. Plesk also did not support PHP5 at that time. I eventually switched over to CPanel and its clunky interface and directory naming, but most of functionality seemed to work much better out of the box and I was able to upgrade to PHP5 via the control panel.
As an aside, thanks for maintaining such an informative blog. I’ve been reading for a few months now and truly enjoy the variety, depth and timeliness of your articles. Now if I can just find a way to get started using the Printable CEO … :)
Richard: I haven’t had any difficulties with Plesk other than it not synchronizing with the mail server once. I’ve never used CPanel before. I haven’t looked into PHP5 and other such things (so far, I’ve had no need of them).
Thanks for the compliment too! It’s much appreciated :-)
Funny enough, I got to this page by Googling “futurequest sucks” too. It fares a lot better than Yahoo webhosting, IX Webhosting and HostGator. I guess I’ll be Futurequesting soon too… thanks David.
Dave,
Even if your article was posted 2 years back it still remains as a very useful resource. As for the hosts mentioned googling “Futurequest sucks” still doesn’t show any relevant results and Media Temple remains as the next soultion to Futurequest’s limitations.
Thanks for your honest reviews.