If you are hosted at ThePlanet or use a service hosted at ThePlanet, then you know the last few days have been tough. In case you missed it, they had an explosion in their transformer area that knocked out power to 9000 servers. Most servers are back online but there are still repercussions from the event. Here are some tips for getting through this event.
DNS
DNS continues to be an issues. The NS# nameservers were offline and continue to have intermittent outages. ThePlanet has setup secondary systems but they are having zone propagation issues.
We've had many clients with email or domain problems due to DNS though their servers were not impacted by the outage. Given that they are still having issues fixing the zone servers:
I highly advise that you switch DNS to outside of ThePlanet.
DNS Options
As noted in my early post about ThePlanet outage, there are many DNS options.
Third Party
If you have a few domains to get going, I suggest you sign up with DNS Made Easy. There are other 3rd party providers like ZoneEdit, but I prefer DME's approach and interface. With a business account you can set very low TTL's in case you want to move domains again. You can get a business account for $60/year and it will handle 25 domains.
DNS Server
If you have 100's of domains, DME may not be cost effective. You may want to consider leasing a server elsewhere, such as SoftLayer, LayeredTech, Bocacom, Ripplewebor similar companies. Make sure they are not ThePlanet resellers.
You can then start building up your DNS records. 4PSA makes a very good DNS manager, though I am sure there are free ones. Also, being handy with a scripting language can be useful. For example, using bash and MySQL's "replace" function, you can quickly take a template and populate it from a list or database to generate your zone files.
Registrar
Many registrars provide custom DNS services. Some may charge a fee others may provide it for no charge, but it is worth checking into.
Email
If you are still seeing outages on your server, I suspect email is one of the key issues. There are several options depending on how you want to handle your mail. Pretty much any of these workarounds will require changes to your DNS. You will need to update your primary record or add a secondary MX record.
Secondary MX
You can setup a secondary mail server pretty quickly and easily. Using your favorite MTA, you can have it up and running in about 30 minutes. It will just spool email for you. This does however require you update your DNS with a secondary MX record. As it is only spooling, a single server can handle 100's of domains.
Catch-alls
You can setup a new server with catch-alls on each of your domains. This will catch email and dump it into a standard mailbox. You can then provide webmail access to the domain admin. This of course has all sorts of privacy issues attached but may be able to save some of you. You could even setup a forward to a third party account like Gmail if you wanted.
Full Email Server
This will require you to have your usernames. You can setup a server just to handle email. Depending on the number of accounts there will be different approaches. You can then have it just process the mail for you and provide your users with their account info.
Backups
If your server is flapping, then try to grab a backup. If you have a lot of data, you may need to compress it first or use the built-in backup tools. If you have just a few gigs of data, rsync can be very handy. Use it with screen to quickly backup your files and resume backups should power cut the transfer.
Databases
I am most familiar with MySQL. If you are running MySQL, I suggest making both file level and SQL dumps. To make file level dumps, just stop the mysql server and tar up the /var/lib/mysql (or wherever your data is stored). Use Bzip to compress and transfer offsite. I recommend bzip as it has some recovery options for corrupt archives.
Server Details
I suggest you dump your Apache, PHP, and kernel version info into a file and save it. Also make backups your your apache and php configs should something happen. If you have any customizations from your server, grab copies of them while it is up and running.
Long Term
In my experience, generator power is not 100% reliable as has already been seen this morning. If your server is on the first floor, I would consider having ThePlanet move it to H2 or you migrate it out of that facility as soon as possible. They are looking at days or weeks to get this fixed. Migrating to a like server will be easier than switching control panels or OS's.
Good Luck
I really do not relying on luck to keep servers running, but in this situation some luck may be needed. We had our share of bad luck yesterday when thunderstorms knocked out our office power while helping people at ThePlanet. Fortunately, we've a number of over-sized UPS's that kept us running until power was restored.
