I recently had to move my website to a newer server on Media Temple, which is quite an undertaking. Even though I was moving within MT’s (dv) line, from 3.5 to 4.0 using their automated migration tools, it’s still a journey fraught with pitfalls and hidden gotchyas. In addition to my own website, there are several websites I maintain for friends, so it becomes a logistical headache. The goal is to move from the old server to the new server with ZERO disruption to any of those websites.

My Server History

I’ve done this type of move before between other web hosts and other servers, usually when I’ve run out of resources due to traffic. It’s not such an issue if you just have a few simple HTML pages, but if you’re running WordPress, you’ll hit the limits much quicker (today, I’d guess it’s around 2000 pageviews a day). You’ll know when your web pages start loading really slowly, or if your hosting provider starts telling you that you’re hogging too many server resources and are starving everyone else. I got kicked off my last shared host sometime in 2007 for a bad plugin that counted downloads by loading THE ENTIRE DOWNLOAD into memory, which brought the server to its knees. As a result, I was offered a choice between paying $200/month for a dedicated server, or moving. That’s when I moved to Media Temple, which is a web host that I’ve wanted to try for some time because they had a reputation of being designer-friendly (and they had cool graphics on their website :)

Because my website was part of the 9rules Network, which had a hosting relationship with Media Temple, I was able to get into a dedicated virtual server setup for a bit less than $50/month. While this was more expensive than the $29/month I had been paying before on FutureQuest’s impeccable servers, it also gave me the equivalent of my very own small server that wasn’t shared with anyone else. This is actually a trick; the (dv) is a virtual dedicated server, as opposed to a physical dedicated server. Through low-level software, one physical dedicated server can be split into multiple virtual servers, which has several advantages. I could go on and on about this, but the general benefit is that you can get a lot of control for a little more money. And, if you’re clever about how you configure your virtual server, you can achieve high performance.

Anyway, I’ve been using Media Temple’s (dv) service for quite some time now, and while there are other options like MT’s (ve) servers (barebones servers that you have to configure from scratch), their grid server cloud system (a form of demand-based cloud computing), and even Amazon’s Elastic Cloud thingy, I’ve stuck with the (dv) because they offer a good blend of direct control and convenient configuration through their control panel. The (dv) servers are configured with the software to serve web pages out-of-the-box, handle customers accounts if you’re doing hosting, and create multiple websites and domain names.

My (dv) 3.5 service is being retired this year, and so I’m forced to move to the newer (dv) 4.0. Exciting, and also daunting! There are newer versions of server software available, and the configuration has changed here and there, so I need to make sure my old setup works on the new server. With 20-odd websites to move, some of them quite complex, I allocated an entire week to carefully move and test each one. And it ended up taking the entire week. I made some configuration changes to make it easier in the future, and have made extensive notes here so I’ll be better prepared for the inevitable future upgrade to (dv) 5.0.

Preparation

There are two guides that were helpful in getting my head wrapped around the entire enterprise. The first is Media Temple’s official migration guide and tips on migrating from (dv) 3.5 to (dv) 4.0 articles on Media Temple. This is what I generally followed. The second resource is Jeff Starr’s (dv) migration experience, which per his usual standard of excellence is detailed and insightful.

Offloading Services

The more services you run on your server, the bigger the headache when it’s time to move. I long ago had dropped the following services from my (dv) for my own sanity:

• mail – Handled by Google Apps for Domains for the past 4 years
• spam filtering – When handling your own mail, this crushes your server because of the huge volumes of spam. With mail gone, so goes spam filtering.
• DNS – Media Temple already gives you access to their domain name server through Account Center, so there’s no need to run this service on your machine

Since these services are already configured elsewhere, I don’t have to move them. However, there is one last service that I till was running, and that was the version control software Subversion aka SVN. After consulting with various people, I decided to dump SVN and upgrade to another system called Mercurial. I converted to Mercurial, and then set up my central repository on bitbucket.org so they hosted everything. The result? One less thing to move.

Establishing Procedures

The general order of operations for moving a website from one server to another is as follows:

1. Copy Files from old to new server
2. Copy Databases from old to new server
3. Configure the new server so it recognizes and responds to your domain name
4. Test locally, resolve any errors
5. Update the Domain Name Server to point to the new server
6. Test again
7. Repeat for each website

There is a built-in “migration tool” that handles everything except for testing and updating the domain name server, but it doesn’t always work because the configuration parameters don’t exactly match between the old server and the new server, and this sometimes breaks the migration process. It tends to work for simple websites, but not for customized setups like mine.

Planning the Site Migration Order

Because I had a couple dozen sites to move, I decided that I’d test the migration tool on the simplest ones. This covered about half the websites, which used very simple HTML and no PHP or fancy server features. For the more complex sites, I would move them one-by-one.

To prepare to move these sites, I had to go to the into the Media Temple Account Center and edit the DNS Zones for each site I planned to move, lowering the Time To Live (TTL) for entry. This values tells users of the Internet that the domain name (e.g. davidseah.com) is going to be changing much more rapidly than usual. The default value is 12 hours, and lowering TTL reduces it to five minutes. You can think of it as telling the Internet to not assume that “davidseah.com HQ” is at a fixed server address, but is about to move. If you don’t lower TTL and point your domain name to the new server, it will take 12 hours for the change to be noticed, and a lot of traffic will go to the old server for that period. It’s a pain in the butt for testing, so that’s why I lower TTL first.

Ordering and Prepping the New Server

You’ll need to have your old server active in addition to the new server. That means you’ll be paying for both servers for the period of time that you are actively moving your websites from one to the other. The Media Temple Guide mentioned earlier details the process; it ends up not being very expensive if you can move your sites quickly.

After you have the new server, immediately request installation of developer tools and shell access. While developer tools may not strictly be necessary, they’re handy when you want to install a few additional packages. I try to keep my server installations are vanilla as possible to keep things simple, but at very minimum you’ll want shell access through SSH, and maybe a favorite text editor if you don’t like the default-installed editors nano and vi.

If your server had more than one IP address, you can contact support and request it and Plesk Migration Manager will be able to use it. The alternative approach is to go back and change your dedicated IP to shared instead before doing the migration. This will break your SSL during the transition, however, if you had it set up.

Save Old Configuration Data

Because I’d customized my old server, I made copies of several key files, just in case I needed to check something after cancelling the old server.

• /etc/passwd, /etc/group – to make sure I didn’t forget any special users or groups that I had created
• password files for .htaccess authentication
• php.ini, my.ini, and other initialization files for important services, so I could compare them to the new server setup

Doing the Migration

The Plesk Migration Manager is a great help in transferring files, account names, databases, and passwords. It also will transfer the contents of /root and all stats and logs that are stored within each domain’s statistics directory. Here’s what it looks like:

Here’s the (dv) 4.0 fancy new control panel! I have highlighted the migration manager entry.

The choices for migration are simple. When you first start, you’ll get the choice to start a new migration. If you have some migrations going on in the background, you’ll see them listed here, as they can sometimes take a long time. In my case, a big site with about 3GB of data took about 45 minutes.

To start the migration, you need to provide the root access credentials to the old server. You can use any domain name that’s on that server; the migration manager will be able to see all websites created with Plesk. I also only chose to migrate parts of the server at a time (see check boxes toward the bottom); I left everything else mostly alone.

The next screen gives you the opportunity to choose which domains or clients to move. I zapped out the other websites in this listing since I don’t think people need to know everything that’s running on my server.

When you first set up a new (dv), you get one IP address that’s shared between multiple domain names. The other option is to have a “dedicated IP” address, which is assigned to one and only one domain name. This is required for security (SSL) certificates, so I requested an extra IP address before starting the migration as I needed to match my old configuration. This allowed me to assign them appropriately on this screen. NOTE: this deviates from the recommended installation instructions, and in my case was necessary because my old server’s Plesk access has been completely broken for months, and I couldn’t change it if I wanted to.

Here is the aftermath of moving davidseah.com. I had an error related to the configuration file, which I was able to fix by going into the davidseah.com vhost.conf via SSH and figuring out that I didn’t have DAV installed on the new server. I just nuked the offending settings, and clicked the link, and the migration continues to completion.

When the Migration Manager Fails

About half the sites I moved, including some very simple ones, had some small configuration glitch. My methodology was to keep two ssh windows open, one connected to the old server and the other to the new server, logged-in as root. If I suspected a discrepancy, I used tar to package up all the files in httpdocs of that site, then used SFTP to securely copy the tar file to the new server:

on old server as root

cd /root
tar cvzf davidseah-files.tgz /var/www/vhosts/davidseah.com/httpdocs/*
sftp {ip-address-of-new-server}
sftp> cd /root
sftp> put davidseah-files.tgz
sftp> quit

switch to new server as root

cd /root
cd /var/www/vhosts/davidseah.com/httpdocs
rm -fr *
tar xvzf /root/davidseah-files.tgz

Be careful with that rm command…it deletes everything in the current directory! Make sure you’re in the right subdirectory!

The migration manager generally handles the creation of users and copying of databases without any problems, except in one case when the database was so large that MySQL timed-out and reported an error. In this case, I then used mysqldump to make a backup, used SFTP to copy the backup to the new server, then used mysql to restore the backup, more or less as follows:

on old server as root

cd /root
mysqldump -u{dbuser} -p {dbname} > dump.sql
sftp {ipaddress-of-new-server}
sftp> cd /root
sftp> put dump.sql
sftp> quit

switch to new server SSH window as root

mysql -u{dbuser} -p {dbname} < /root/dump.sql

General Observations on the Plesk Migration Manager

• It tends to choke on converting domain account settings and permissions from the old Plesk 8.0 in dv (3.5) to Plesk 11. This will require manual tweaking and some knowledge of both Plesk account models. The new thing in Plesk 11 is the notion of “subscriptions”, which didn’t exist in 8.0. If you have multiple domains and customers before, then expect to spend some time defining basic levels of service such as disk space, bandwidth, and other server features. You can override all this stuff for now, giving everyone lots of access to everything, just to make sure the sites work.

• Plesk Migration Manager useful for moving MOST files, including everything in each domain’s private directory, but not all files in the conf directory (if you have added custom files there, as I did for user authentication with htaccess, they won’t be copied. Users, including web users, were transferred. Groups, however, were NOT. Anything you added to your /etc directory isn’t copied. Plesk Migration Manager will NOT move anything that’s not part of the standard Plesk setup.

• For very large database transfers, Plesk Migration Manager can fail with a “MySQL server has gone away”. This might be due to a timeout error. In these cases, I just used mysqldump and then used SFTP to grab the file directly between servers (much faster than downloading and reuploading 2GB of data), then restoring the dump.

• The same server-to-server SFTP can be use to move files. If you use tar to create an archive of all the files in your httpdocs directory, this is fast and reliable.

• If you have added fancy Apache directives in your vhosts.conf file that depend on modules you added yourself, then you may get Plesk migration errors if you didn’t add them on the new server. For example, I had installed WebDAV on my old server, and the Migration Manager complained bitterly about misconfiguration until I nuked the related directives out of the afflicted vhosts.conf.

Other Critical Server Differences

I found that the (dv) 4.0 has a few critical differences that I had to understand before I could get my PHP and MySQL databases fully functioning on all sites:

• I had been using mod_php before, which requires that any directories that your PHP scripts needs to write (for example, cache files or image uploads) be owned by the apache user. Since the directories are usually owned by a different user, this creates something of a pickle: you have to essentially leave your front door unlocked on several directories; on my old server, I’d added the apache user to the psacln group by modifying the /etc/group file, which is sort of like installing a pet hatch in your front door.

• FastCGI PHP, unlike mod_php, has the advantage of not requiring writeable directory hacks. Instead, PHP runs under your domain’s user account, so you can just have them all owned by that user. However, Plesk Migration Manager will retain your old file ownerships…you MUST change their ownership with a quick chown -R username.psacln * executed inside the httpdocs directory, if you previously used mod_php instead of FastCGI. This is described in this MT knowledge base article.

• The nginx reverse proxy server was unfamiliar to me, which is another way of serving web pages to the Internet. It’s supposed to be REALLY fast, but as configured on the (dv) 4.0 out of the box, it is not compatible with WordPress Networks AKA Multisite. I just disabled the service, letting Apache take on its traditional role as main web server. There may be performance repercussions, but it was just easier for me to disable nginx until I had the time and inclination to study it. The problem seems to be that it doesn’t handle “wildcard IP address” assignment in the Plesk control panel, and it also breaks the ability to set a default website for each IP address you have. This is a known bug at this time.

• The default PHP security settings have the OPENBASEDIR option set to “default”, which can be a pain in the butt for some scripts. If you see errors where a script cannot open some file “above” it, it’s probably this. Set it to “none” or figure out what paths need to be added for extra security. My understanding is that OPENBASEDIR is not really a security panacea to begin with.

• The (dv) 4 hardware platform is a 64-bit architecture, compared to the (dv) 3.5′s 32-bit architecture. This has ramifications for some MySQL database types, so if you are relying on max/minimum sizes for unsigned ints, they have changed. This affected my realtime web analytics package, Mint. Fortunately there was a fix provided. WordPress was unaffected.

• The new Plesk makes some operations, such as adding options to php.ini, a little different. For example, instead of creating a custom vhost.conf in your /var/vhosts/yoursite.com/conf folder, you can add them to the Website Advanced Setting / PHP Settings in the “Custom Directives” box. This is more Plesk friendly, and I imagine it will migrate with you in future moves.

Adapting to the new Plesk

As I mentioned earlier, Plesk 11 adds a new “subscription” model layered on top of the familiar domains and customers. Basically, a subscription a set of server presets that’s assigned to each domain. For example, the included “Simple Website” subscription specifies 100MB of disk storage, 1GB of network transfer, and the ability to use PHP.

Since subscriptions don’t exist in Plesk 8.0, the Migration Manager either can jam each domain into an existing subscription, or it can just assign a “custom setup”. You have to play with the website setup for the domains to get a feel for what’s available, and consult online Plesk documentation to figure out what each option does. This also requires a fairly advanced understanding of web server technology for it to make any sense.

• You can create a number of subscription profiles and then assign them, though it’s not necessary. In my transition, I assigned custom setups during the initial transfer, and later created subscription packages that made sense for me.

• Beware use of “the expiration date” feature, because Plesk will calculate the expiration based on the expiration + the original account creation date. For example, if your client example.com was created on September 15, 2005, and you set the length of the subscription to 1 year, it expires on September 15, 2006. You won’t see this until Plesk does its 2AM accounting, and everyone will get a “your domain has expired” email, which is very hostile. I just set everything to unlimited for now.

• Plesk now has three control panels instead of one. The main ones are at https://yoursite.com::8443, and the “Power Control Panel” which manages the nuts and bolts of your server is located at https://yoursite.com:4643/. You will need the admin password you created when you first ordered the new server for the regular Plesk panel, and your root password for the Power Control Panel.

• New Copy Database feature, great for cloning databases for quickbackup. However, be warned that a cloned database does NOT clone users; you’ll have to recreate them.

Previewing your Server Changes Without Changing Main DNS Records

Once you’ve moved your files, databases, and configurations for a particular domain to the new server, you have three ways to test it:

1. You could go to the Media Temple Account Center and change the DNS for the domain so it points at the new server IP address. However, if it’s broken it will break your website for all visitors.

2. A neat alternative trick is to edit your own computer’s local hosts file so only it knows about the new server. This overrides the DNS that the world sees with your new value, which allows you to see the site as if it has already be transitioned. Everyone else will be unaffected. If everything looks good on your browser, then go to option 1 and change it for real.

3. The Plesk Control panel has several forms of site preview, but I can never find the icon for it and it doesn’t work very well. Go with option 2.

Editing your host file on the Macintosh or Windows is the least disruptive way. Just remember it works only on your computer, and that you should remove your changes once you’ve transitioned your website completely to the new server and made the DNS switch. Be warned also that some browsers do additional caching of DNS entries (Chrome, for example), so you will need to disable “DNS pre-fetching” or “network optimization” in the advanced settings, somewhere.

Optimizing and Tweaking the Server

I disabled a few services from autostarting, because they’re not needed. This performance tuning article is helpful in getting started.

To disable additional services, you can dig into the etc/init.d directory and chmod 644 any startup scripts you don’t want running. You can also do this through the Plesk Power Control Panel, but I find the chmod approach easier. I disabled the following scripts:

• named – this is the local nameserver service, which is totally unnecessary
• spamassassin – since I don’t receive mail on this server, I don’t need this very resource-hoggy service running
• psa-spamassassmin – this is the Plesk version of spamassassin, which I didn’t notice before
• nginx – this is the front-end web server

Before applying the chmod 644 to these scripts, you first want to shut them down (if they are running, which you can check with a ps aux or by running top) by typing:

    /etc/init.d/named stop
    /etc/init.d/spamassassin stop
    ...

…and so on. Then, apply the permission change with chmod to prevent them from ever running again.

Simple Transition Monitoring

I let the both sites run for two weeks afterwards, to make sure that everything was working properly. By keeping the old server up, I had the ability to go back and check its configuration files to see if there were any differences.

I also kept top running in a terminal window during the transition test period, watching memory usage and looking for anything weird. Here’s an example of what I look for:

The yellow line at the top shows the overall memory usage. The number that concerns me is the memory free line, because if it dips below 50MB (it’s around 940MB here) then the server is in danger of running out of working memory, which will result in a massive slowdown. The key variable that contributes to memory use is the number of people browsing webpages on the server, which includes not just visitors to davidseah.com but also the other 20-odd websites.

I’ve highlighted the three main users of memory on my system:

• Purple shows the number of Apache webserver processes. Every time a web page or image is requested, there is a “httpd process” that is serving it. Each httpd process takes up about 20MB on average, and the number of them is determined by how many requests are active at the same time. For a site like mine, not very many people are hammering my website unless some larger news site links to it, so this “peak usage” must be accounted for if I don’t want my server to go down during extreme traffic events.

• Green shows the memory of of MySQL processes, the database server that helps power the dynamic websites on the server. These numbers don’t change often. My server actually runs two MySQL instances, because I’ve isolated the davidseah.com database from the other sites to see if it helped performance. It doesn’t really, so I may switch back to the single instance. Here you can see around 245MB is used by both MySQL instances.

• Blue shows the memory use by PHP running as a FASTCGI module. Each PHP process runs PHP scripts that power dynamic web pages with MySQL, and this number rises and falls as well with the number of simultaneous web page requests at a given time. It’s closely tied with the number of active Apache processes (purple).

There are some other memory resources that aren’t shown by top (interprocess communication buffers, for example, which is a limited resource). You can see those by going to the Power Control Panel and looking at the Quality of Service (QoS) Alerts.

To tune my server, I need to check a few configuration files:

• The Apache Configuration, particularly MaxServers and MaxClients settings. In the past I’ve found MT’s recommended settings to be overly aggressive, failing completely under extreme server load, so I use my own recommendations to avoid exceeding my guaranteed memory allocation. Running out of memory, at least in the past, exactly a much greater performance penalty than making a web browser wait a few seconds more. I have yet to really test this, though. The pertinent configuration file is at /etc/httpd/conf/httpd.conf; don’t forget to /usr/sbin/apachectl restart or /etc/init.d/httpd restart to make the changes take effect.

• The number of FastCGI PHP processes can be limited as well, though I haven’t had any practical experience tuning this. The pertinent configuration file is located at /etc/httpd/conf.d/fcgid.conf. This article on fcgid installation and tuning is worth studying.

Tuning is an ongoing process, and optimizations at the website and software level can yield large gains in performance. What I’ve described here is the basic approach: know how much memory your critical web server processes are taking up, and make sure that they don’t run out of memory in extreme conditions. You can slim down the processes themselves with clever configuration and other software optimizations.

Back to Business

After working out the aforementioned issues with server configuration differences, everything is working smoothly, and I’m ready to turn off the old server and let things go. I really do like the (dv) setup. There may be a few extra configuration tweaks to make; it might be nice to learn how to configure nginx to see if I can lower the average memory use of the server, but in the meantime everything appears to be working fine without it.

The (dv) 4.0 continues to offer me a good balance of cost, performance, control, and ease of configuration…the next step up is a dedicated server, or perhaps some cloud-based approach. There are a lot of other companies offering similar hosting packages that I’ve been intrigued by, but they seem to be smaller players and I don’t really want to change my hosting if I can avoid it…as you can gather, it’s kind of a giant pain in the butt! I’d rather just let it run and not think about it :)