IT outages happen to everyone

For the past few hours the internet messenger Slack has been down. A few weeks ago you couldn't use Google's services, and even earlier a large part of the Internet wasn't working because of an outage of Cloudflare's services. Is it possible that cloud services are unavailable?

The unavailability of an IT system is a natural thing and we should get used to it instead of believing the marketing assurances that cloud services work non-stop.

Every broken piece of technical equipment works correctly in the presence of qualified repair personnel.

We have to be prepared for things like outages in server rooms, floods or fires, which can destroy our entire server infrastructure.

Why are IT systems unavailable?

A website doesn't have to be available all the time, and what's more, there are many reasons for it not to work. Updating the software of the site itself or of the server infrastructure is one of the main causes of planned unavailability. Breaks of this kind aren't incidents; in fact, they're even something desirable.

Another cause is incidents which, despite ensuring a high SLA in the infrastructure, allow for a certain amount of unavailability. These are all sorts of failures related to bugs in software, increased and unforeseen traffic on the site, or damage to elements of the infrastructure. The source of these incidents is problems.

Relax, it's just an outage!

Firefighting, that is, resolving incidents, is a constant element of working with IT systems. In theory you can prepare an IT system that's resistant to incidents and problems, but the costs of implementing and maintaining it will be astronomically high.

From a practical point of view, a better solution is to estimate and accept the losses associated with an incident in relation to the costs incurred to avoid that outage. An example of this could be an online shop running on a hosting server whose annual cost is 200 zł. If we earn 50 zł a month on such a shop, then a server outage lasting over an hour is cheaper than buying servers with higher availability.

However, if the shop has a turnover of around 10,000 zł within an hour, then downtime lasting even 30 minutes is costly for us. In that case a good solution is to buy much more expensive server infrastructure with increased availability.

You'll learn more about high-SLA systems from my talk at the PHPCon Poland 2019 conference.

Incident or problem?

Outages can be divided into incidents and problems. This classification is adopted in ITIL (Information Technology Infrastructure Library). Even though this set of practices was devised in the 1980s, it still applies to today's reality.

An incident is an unplanned interruption of the delivery of a service or the malfunction of a component that hasn't yet affected the service. A failure of a payment terminal in one of the shops during its working hours is an incident. There's an interruption in the delivery of the payment service. If such a situation occurred outside the shop's opening hours, it won't be classified as an incident.

Likewise, all sorts of planned work, maintenance work during service hours, aren't classified as incidents, because they aren't unplanned.

Incidents should be resolved as quickly as possible, regardless of whether we apply fixes or a temporary workaround.

A problem is the cause of one or more incidents. The goal of outage management is to identify problems so that they don't generate even more such events in the future.

Referring back to the example of the malfunctioning payment terminal: the problem was a bug in the device's software, which caused it to hang. The temporary workaround was restarting the device, and the fix was a software update. Identifying the problem and applying the fix guarantees that similar incidents won't appear in other shops.

Outages at the biggest players

Incidents or planned unavailability happen to everyone, even the largest provider of IT systems. Each time, they report problems related to their services on special websites called status pages.

A good practice after every outage is to describe what caused it and how such problems can be prevented. Cloudflare describes this very interestingly on its blog. An interesting approach to an outage was also GitLab's live streaming of the system's repair work.

Fire at the OVH data centre

An example of a major outage could be the fire in the SBG2 block of the OVH server room in Strasbourg. It also damaged the SGB1 block, while the SGB3 and SGB4 blocks were shut down for safety.

Hosted on the burnt servers were over 3.6 million websites, including niche government platforms in France, the United Kingdom, Poland and Ivory Coast.

Companies keeping their backups on the same server or in the same data centre irretrievably lost their data. That's why it's so important to store server backups in a completely different and independent location.

Summary

Regardless of whether you're a large or a small provider of IT-related services, you have to prepare for incidents and make your clients aware of this. Preparing an outage report, tracking down the problem and describing the procedures to protect against its recurrence is downright mandatory.