"My clients are responsible for backing up their sites, so I don't need to bother with them." We hear this often. Unfortunately, this stance is often taken for legal reasons and not technical ones. Limiting your liability for the loss of data is crucial for almost any web hosting operation; however, there are other important reasons why you should backup irrespective of your customers' data. The time required to fully restore a server and notify your clients of their new access information can easily cost you more than deploying a backup solution for a year.
Every month rackAID assists people with recovering from some type of system failure. Whether it is a security incident, failed hardware or corrupted software, access to off-server backups, in nearly every case, significantly reduces recovery costs. Maintaining backups is essential to long-term operational and financial stability. Many costs are hidden and not considered until after the fact.
Below, we present a real-world horror story to emphasize the cost of not backing up your server. Consider how your business would fare in a similar situation. We are not trying to put you in a panic or bore you with numbers, but sometimes a real world example illustrates a point better than rhetoric.
Now on to the horror story...
The Incident
A small web hosting company had about 250 domains hosted on a server - one of four servers that they operate. About 5 PM one day, the server died. Reboots failed, emergency recovery tools failed, a full system restore was in order.
After about two hours, the hosting provider determined that the old drive could not be booted. They would get a new server ready and mount the old drive as a slave drive. Around 9PM, they had this ready and we got to work.
The Problem Worsens
Initially, we tried to access the data but the old drive had a critical failure. Often in critical failures like this, we can recover data from the old drive. In this case though, the old drive had a hardware failure. A full recovery from backups would be required. This is when a new problem developed: the company did not have off-server backups.
The False Perception
End users, of course, are unhappy when their data is wiped out by a server failure, but the companies TOS and sign up information stressed that it is the client's responsibility to backup their data. As a result, the hosting company never deployed a server-level backup solution. Like many hosts, they never considered the repercussions of a full server failure. They never considered how much it would cost them just to recover a system to a point at which their clients, if they had them, could restore backups.
The Recovery
So that you learn from their misfortune, we pass on to you what had to be done to get them operational again.
What the web host did not anticipate the amount of time it would take to manually reconfigure some 4 different hosting packages for 250 domains and notify the end-users of their new passwords. We see this very often. Hosting companies start out with only ten or twenty domains on the server, which would not take too long to setup, but as they grow they forget to reassess the impact of a complete server failure.
Fortunately, this company had a spreadsheet with some client data which contained domain, username, email, and service plan. Using this information, we were able to develop a custom script that would the setup the sites with this information, this saved hours of manual data re-entry. To complicate issues, many of the clients had provided an email address that used the domain on the crashed server, so the company still had to make over 75 phone calls to inform clients of their new login information. Also during this period, they received 100's of emails and helpdesk tickets that had to be resolved due to slight configuration and version differences between the new and old servers.
The manual restoration took about seven hours in addition to the initial four hours taken by the data center. Recovery would have taken nearly twice as long if they did not have their spread sheet data. Despite their TOS and warnings, not surprisingly, many clients did not have backups or only had web site backups not email backups. As a result, they lost several clients.
The Costs
In direct costs, the hosting company had spent nearly $400 for emergency disaster recovery services, $600 in overtime for their staff, and lost nearly $100/month in hosting revenues. They also provided refunds to a number of clients, which cost them another $100. This single incident cost them $1100 in one time fees and $100/month in lost revenues. Estimated annualized losses were more than $2300. For $2300, they could have leased a budget server for nearly two years and just used it for backups alone! These direct costs do not even include the intangibles such as a damaged reputation and clients that will leave 2-3 months later due to the incident.
Now consider the same scenario if the company had off-server backups. The data center could have restored the system in about two hours if the special request of mounting the old drive was not made. Then the restore process would have taken about four hours total but only three hours would be hands-on work. The emergency services bill would have been closer to $200 instead of $400, their staff could have gone home on time, saving another $600, and they may not have lost as many clients since the total outage would have been less than 3 hours for some clients and 6 hours at most. Since the backups contained important customizations, they would have saved even more time and money due to fewer helpdesk tickets. With backups, this same incident would have resulted in $200 in direct losses instead of $1000. The company would have likely retained many more clients since most would have had service restored within 4 hours. If they had cut their revenue losses to $25/month, their estimated annualized losses would have been $500 instead of $2300. If you do the math, you see that this leaves them $1700 for backups. With backup solutions as low as $20/month at some providers, you can clearly see that it costs more not to have backups.
The Lesson
For most hosting operations, maintaining backups is a good business decision. Even if your TOS limits your liability due to the loss of data, you still have to get your server up and running. The faster you can return to operations the less money you lose.
For most hosting companies, the revenues from just two or three hosting clients will likely cover the costs of implementing a backup solution for that server. You do not even have to notify your clients that you make backups. You are not backing up the data for them you are backing it up for yourself. In fact, you may not even want your clients to know you maintain backups. Doing so may make them reliant on you for backups despite your TOS.
The cost of maintaining backups is trivial compared to the cost of server failure. Even non-hosting companies with complex or large sites could incur hours of downtime if you do not have backups. Consider the case above: How much money would you lose? Now, can you afford backups?
