This morning I was sitting in the waiting area waiting to get some blood taken for a routine old bastard test. All of a sudden my phone started going off with SMS messages advising me multiple web servers and sites were down. Crap!
A quick check at the network status of the upstream provider confirmed there were some network issues, so I started notifying clients and began to prepare to execute their failover plans to keep everything on the air.
Luckily service was restored within about 30mins and by the time I got to my desk everything was back to normal. Phew
However it reminded me of a hard lesson I learnt about five years ago when working at a company whose primary source of income was online sales from three different websites.
It was early December, and the company’s websites went down. All three revenue streams, down.
Initially we weren’t too stressed, thought it would just be a temporary glitch. After all we had outsourced the web hosting to a reputable company and were paying a pretty penny to do so.
However it dragged on for twelve days. The hosting company eventually stopped answering phone calls and attempts to get in touch with them failed. Bugger
Eventually service was restored just in time to salvage some sales, but we were way down on target, only just broke even, and went into the nororiously quiet start of the new year without the buffer that a good December usually provides.
The lesson learnt was quite a simple one really, Don’t put all your eggs in one basket.
This lesson led me to schooling up on cloud computing and studying for the AWS Solutions Architect certification from Amazon Web Services.
Where were all the eggs?
Let’s just back up a bit for those that may not be too techy and have a look a the things that go together that make a web site work.
At its basic level for an online presence you need.
- DNS. Like the internets telephone directory, you need an entry on a server that tells people looking for www.yourawesome.site.com.au that it lives at IP address 220.127.116.11
- The Web Server. To store and serve up your web site.
- An Email Server. Because you want email, right?
Old school web hosts and even many still about today host all of these services on a single point of failure. An outage will typically affect your entire web presence.
Why is this particularly bad?
Where things can go really pear shaped is that often your backups will be stored on the web server, so unless you have downloaded your recent backups on a daily basis and kept them somewhere on your own computer you don’t have access to them.
Chances are you don’t have a local copy of your backups, because right next to the part telling you that the web hosting provies 99.9% uptime there was a part that said Backups included.
Backups are useless unless you have access to them.
The other thing that is also a right pest is that if you can’t get at the DNS records, moving the site becomes a much harder task. Not to mention the fact that redelegating a domain usually means having access to your email, which you don’t have because that was bundled into your hosting as well.
So things can go to shite pretty quickly.
Mitigating the risk
So let’s have a look how we can make sure we don’t end up in a pickle when things go bad.
The first thing you need to figure out is how much downtime can you afford? Hours, Days, Weeks? At what point does an outage really start costing you money?
Of course the answer will determine your monthly spend, so be realistic. Think of it like insurance. The more coverage you want, the higher the premium.
Moderate availability 24-36 hours
This is what I recommend as an absolute bare minimum. It is the most cost effective and the easiest to set up.
In a nutshell it revolves around isolating each of the moving parts and having a seperate accessible place to put your backups.
So let’s look at a common solution.
1. Use a service that specialises in managing DNS.
- Google Domains (https://domains.google/intl/en_au/)
- Cloudflare DNS (https://www.cloudflare.com/en-au/dns/)
- AWS Route 53 (https://aws.amazon.com/route53/)
- DNS Made Easy (https://dnsmadeeasy.com/)
are some that come to mind.
2. Use a VPS for your web hosting
VPS stands for Virtual Private Server and is the mainstay of cloud based computing.
Basically it allows us to have our own mini server in the cloud, plus spin up more very quickly if we need to.
Once the realm of the linux nerd, it is easier these days to host your site on a VPS thanks to managed services like :
3. Have a place in the cloud for your backups
Make sure your web server backs up to a place that is not itself. Cloud based storage solutions are extremely reliable these days, and likely safer than storing a backup on your own computer.
Options here include :
- Google Drive (https://www.google.com/drive/)
- DropBox (https://dropbox.com)
- Amazon S3 (https://aws.amazon.com/s3/)
4. Use a dedicated email provider
Much more common these days than it used to be, but people still get suckered into the all in one packages.
Keep your email separate.
- Gmail (Gsuite so you can have @yourdomain)
- Amazon Work Mail
The failover plan
In this set up the thing that is most likely to fail is the web server. The dedicated provider you’ll for your other services have their own redundancy in place.
When did you last see Google go offline? Or Amazon for that matter?
So let’s say we have a failure at the web server level that can’t be fixed easily, it is getting towards our maximum window of pain, what do we do?
- Spin up a new VPS maybe with another cloud provider.
- Restore the website from your cloud backup
- Change the DNS record to point to the address of the new VPS.
That’s it, you’re back online.
I was going to discuss some high availability architecture but it is probably outside the scope of this blog post, and I could end up going on for hours talking about stuff nobody is particularly interested in. Especially not site owners as they just want their stuff to work.
Let’s just say that if you want higher availability in the event of a service failure, or even during periods of high load (you are running a big promotion, or a blog post goes viral) there are ways to set it up.