This article applies to an older version of the Media Temple (dv) service from 2007. You may want to check Media Temple’s knowledge base for updated information.

I’m running a (dv) Base system from Media Temple, which is their 256MB version. This is their lowest-end server configuration, which is actually a virtual server sharing the same hardware.

As my blog has grown in popularity, when I switched over I got enough traffic to overwhelm the server. It turned out that the base configuration isn’t tuned to work within the guaranteed 256MB of memory allocated. It’s configured to run on a considerably larger server, and grabs extra memory as needed if it’s available from the global memory pool. This is all fine and dandy when memory is available, but if it ISN’T, the server will start to chug.

After making the following tweaks to my server setup, one of my articles was dugg and the server was hit by 2750 pageloads per hour for a few hours. This is a pretty significant amount of traffic, and the server had no problems staying up. In fact, it barely broke a sweat. Keep in mind though that in addition to the server tweaks below, I also had optimized my wordpress installation with WP-Cache, which reduces server load significantly. You can read more about this in WordPress and Shared Hosting.

Anyway, here’s what I’ve done to get my DV in line. I’ll write about the WordPress issues a little later. If this article convinces ya to get a (dv), refer my site when you order so I get a little reward from Media Temple :-)

Check for Memory Faults

The virtual server is a product called “Virtuozzo”, and there’s a special file that tells you how much memory you’re allowed to use. When you exceed that amount of memory, a fault is generated. Ideally, you don’t want any faults.

To check up on this, you can use Plesk (under Virtuozzo) or login to your server and type cat /proc/user_beancounters into the command shell. The column that’s of most immediate use to you is failcnt, which tells you how many times a program has tried to get some memory to do something, but wasn’t able to do so. The values should all be 0.

I’ll fill this section in a little later, because it’s kind of tricky to understand and I don’t entirely get it. Here’s a cheesy script I use to constantly monitor the user_beancounters to see if they’re going up (I just open a second shell window):

#!/bin/sh
while true ; do cat /proc/user_beancounters ; sleep 1 ; done

When you run this script, it just cats the output every second (see image). If you’re monitoring other parts of the server, you might see some overall patterns in what seems to be causing the memory usage. There’s some more information at the very end of this post (with screenshots) to help orient you.

Apache Configuration

The default configuration for Apache allowed too many instances of httpd to spawn. Since each instance takes between 20MB and 40MB, you don’t want more than your available memory to be created. If I figure on allowing 128MB available out of my 256MB, and each Apache instance takes 20MB, that means I can’t have too many of them active at once. In actuality, the memory usage reported by ps aux under the VSZ and RSS headings isn’t the real picture of memory usage (see this article for some insight into this), so I can actually have more instances. I experimentally determined that 20 seemed to be enough without pushing the server too hard.

The pertinent settings are in the prefork section of /etc/httpd/conf/httpd.conf. Make sure you save a backup of the file. Here’s what my prefork section looks like (SAVE A BACKUP BEFORE YOU EDIT!)

<ifModule prefork.c>
StartServers 1
MinSpareServers 1
MaxSpareServers 3
ServerLimit 20
MaxClients 20
MaxRequestsPerChild  4000
</ifModule>

UPDATE: On the week of August 21, 2007, I noticed that my memory allocation was exceeded regularly by about 100MB or so. It seems that the way (MT) is calculating physical memory use is also taking virtual memory into account. I had to drop ServerLimit from 20 to 6 to maintain responsiveness. I have a ticket open to see if this is a bug or not (virtual memory is disk-based, so you’d think it wouldn’t count against your physical memory allocation). :END UPDATE

UPDATE:Chris McKee points out that Media Temple has a performance tuning knowledge base article with some recommendations for the Apache configuration. My settings are much more conservative, designed to avoid memory swapping altogether.

I also disabled modules that I wasn’t using by commenting them out. I reduced the memory footprint of each httpd instance by a bit. Every bit helps! The lines with the # in front are disabled modules.

LoadModule access_module modules/mod_access.so
LoadModule auth_module modules/mod_auth.so
# LoadModule auth_anon_module modules/mod_auth_anon.so
# LoadModule auth_dbm_module modules/mod_auth_dbm.so
# LoadModule auth_digest_module modules/mod_auth_digest.so
# LoadModule ldap_module modules/mod_ldap.so
# LoadModule auth_ldap_module modules/mod_auth_ldap.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
# LoadModule mime_magic_module modules/mod_mime_magic.so
# LoadModule cern_meta_module modules/mod_cern_meta.so
LoadModule expires_module modules/mod_expires.so
# LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
# LoadModule usertrack_module modules/mod_usertrack.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
# LoadModule dav_module modules/mod_dav.so
# LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
# LoadModule asis_module modules/mod_asis.so
# LoadModule info_module modules/mod_info.so
# LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
# LoadModule imap_module modules/mod_imap.so
LoadModule actions_module modules/mod_actions.so
# LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
# LoadModule proxy_module modules/mod_proxy.so
# LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
# LoadModule proxy_http_module modules/mod_proxy_http.so
# LoadModule proxy_connect_module modules/mod_proxy_connect.so
# LoadModule cache_module modules/mod_cache.so
LoadModule suexec_module modules/mod_suexec.so
# LoadModule disk_cache_module modules/mod_disk_cache.so
# LoadModule file_cache_module modules/mod_file_cache.so
# LoadModule mem_cache_module modules/mod_mem_cache.so
# LoadModule cgi_module modules/mod_cgi.so
LoadModule logio_module /usr/lib/httpd/modules/mod_logio.so

After you make these changes, type apachectl configtest to make sure you haven’t screwed something up. If you get the Syntax OK go-ahead, then apachectl restart will restart the server and reread the configuration.

Note: Your website might actually use some of the modules that I’m not using, so be careful.

Note: ps -A -o pid,vsz,rss,pmem,time,comm shows memory usage (VSZ and RSS) roughly, though it’s not really accurate.

SMTP connections

SMTP is used to receive mail addressed to mailboxes on your server. It’s also used to send mail to other people from your email account.

By default, the number of incoming SMTP connections was set to “unlimited”. When I was getting hammered by spam, this created a LOT of simultaneous connections, each of which took up memory. I limited the number of instances to 10 by editing /etc/xinetd.d/smtp_psa, from the default of 0 (unlimited).

I also noticed that when a secure SMTP connection (SMTPS) occured, the SMTPS daemon would use 100% CPU. I don’t use secure SMTP, so I denied all SMTPS connections to prevent spammers connecting to the secure port from creating this problem. That file is /etc/xinetd.d/smtps_psa.

After you make these changes, type /etc/init.d/xinetd restart to tell the server to re-read and use the new configuration information.

POP3 and IMAP SSL

Since I don’t use SSL for IMAP or POP (these are services that email programs use to download your mail), I disabled these scripts as well by commenting out the lines that start with report_action "Starting imap-ssl" and report_action "Starting pop3-ssl". I will have to look back at these later to see if they really make a difference.

To restart the services, type /etc/init.d/courier-imap restart.

MySQL Configuration

Although I wasn’t sure if MySQL was a problem, I decided to configure it to run in a smaller memory footprint. I’m not an expert in MySQL configuration, so I just used the “medium” memory configuration as a starting point. You can find the my-medium.cnf file, suitable for a 64MB instance, in /usr/share/doc/mysql-server-4.1.20/. The changes I made were in the [mysqld] section of the file /etc/my.cnf

[mysqld]
set-variable=local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1
skip-bdb

set-variable = max_allowed_packet=1M
set-variable = key_buffer=16M
set-variable = table_cache=64
set-variable = sort_buffer=512K
set-variable = net_buffer_length=8K

set-variable = myisam_sort_buffer_size=8M

I basically added the set-variable commands.

When I was importing my WordPress database, I also had to increase max_allowed_packet to 16MB. It turned out that there was an inefficiency in my WordPress setup that actually was the source of this problem: The Tan Tan Reports plugin created a very large entry in the wp-options database table. I removed the plugin and its data from the wordpress database table and I was able to reduce it back to 1MB. This was very important from a memory usage perspective too; MySQL allocates extra memory for inter-process communication (like, between MySQL and WordPress), and this caused memory failures in the user_beancounters!

After you make these changes, type /etc/init.d/mysqld restart to reload the MySQL configuration.

Additional Optimizations

There are some additional things I did, like disabling spamassassin (21MB), limiting the number of plesk connections (another 30MB), and disabling named (1MB). You can read about these additional optimizations over here.

Screenshots

See this page for some screens and descriptions of how I am monitoring memory usage the old-fashioned way.