This morning I was sitting in the waiting area waiting to get some blood taken for a routine old bastard test. All of a sudden my phone started going off with SMS messages advising me multiple web servers and sites were down. Crap!
A quick check at the network status of the upstream provider confirmed there were some network issues, so I started notifying clients and began to prepare to execute their failover plans to keep everything on the air.
Luckily service was restored within about 30mins and by the time I got to my desk everything was back to normal. Phew
However it reminded me of a hard lesson I learnt about five years ago when working at a company whose primary source of income was online sales from three different websites.
It was early December, and the company’s websites went down. All three revenue streams, down.
Initially we weren’t too stressed, thought it would just be a temporary glitch. After all we had outsourced the web hosting to a reputable company and were paying a pretty penny to do so.
However after numerous support requests and phone calls, it ended up dragging on for twelve days. The hosting company eventually stopped answering phone calls, their own website and email services were offline and attempts to get in touch with them failed. Not soo good.
Eventually service was restored just in time to salvage some sales, but we were way down on target, only just broke even, and went into the notoriously quiet start of the new year without the buffer that a good December usually provides. Anyone that relies on retail sales for a living will tell you the pre christmas period is where a good portion of the years sales are made. Jan and Feb can be quiet months as peoples credit card bills start rolling in and they get settled back into the real world.
The lesson learnt was quite a simple one really, Don’t put all your eggs in one basket.
This lesson led me to schooling up on cloud computing and studying for the AWS Solutions Architect certification from Amazon Web Services. Back in those days cloud computing was still kind of new and not really accessible to small to medium sized businesses.
Let’s just back up a bit for those that may not be too techy and have a look a the things that go together that make a web site work.
At its basic level for an online presence you need.
Old school web hosts and even many still about today host all of these services on a single point of failure. An outage will typically affect your entire web presence.
Where things can go really pear shaped is that often your backups will be stored on the web server, so unless you have downloaded your recent backups on a daily basis and kept them somewhere on your own computer you don’t have access to them.
Chances are you don’t have a local copy of your backups, because right next to the part telling you that the web hosting provies 99.9% uptime there was a part that said Backups included.
Backups are useless unless you have access to them.
The other thing that is also a right pest is that if you can’t get at the DNS records, moving the site becomes a much harder task. Not to mention the fact that redelegating a domain usually means having access to your email, which you don’t have because that was bundled into your hosting as well.
So things can go to shite pretty quickly.
So let’s have a look how we can make sure we don’t end up in a pickle when things go bad.
The first thing you need to figure out is how much downtime can you afford? Hours, Days, Weeks? At what point does an outage really start costing you money?
Of course the answer will determine your monthly spend, so be realistic. Think of it like insurance. The more coverage you want, the higher the premium.
This is what I recommend as an absolute bare minimum. It is the most cost effective and the easiest to set up.
In a nutshell it revolves around isolating each of the moving parts and having a seperate accessible place to put your backups.
So let’s look at a common solution.
1. Use a service that specialises in managing DNS.
are some that come to mind.
2. Use a VPS for your web hosting
VPS stands for Virtual Private Server and is the mainstay of cloud based computing.
Basically it allows us to have our own mini server in the cloud, plus spin up more very quickly if we need to.
Once the realm of the linux nerd, it is easier these days to host your site on a VPS thanks to managed services like :
3. Have a place in the cloud for your backups
Make sure your web server backs up to a place that is not itself. Cloud based storage solutions are extremely reliable these days, safer than storing a backup on your own computer and likely safer than storing it in a bank vault.
Options here include :
4. Use a dedicated email provider
Much more common these days than it used to be, but people still get suckered into the all in one packages.
Keep your email separate.
Services like
In this set up the thing that is most likely to fail is the web server. The dedicated provider you'll for your other services have their own redundancy in place.
When did you last see Google go offline? Or Amazon for that matter?
So let’s say we have a failure at the web server level that can’t be fixed easily, it is getting towards our maximum window of pain, what do we do?
That’s it, you’re back online.
I was going to discuss some high availability architecture but it is probably outside the scope of this blog post, and I could end up going on for hours talking about stuff nobody is particularly interested in. Especially not site owners as they just want their stuff to work.
Let’s just say that if you want higher availability in the event of a service failure, or even during periods of high load (you are running a big promotion, or a blog post goes viral) there are ways to set it up.