Availability, Reliability and Fault Tolerance

Concept

These three terms describe different properties, and mixing them up produces designs that miss their goal.

A system can be highly available and unreliable: it fails often but recovers in seconds. It can be reliable but not fault tolerant: it rarely fails, and when it does, everything stops.

Availability, in numbers

A year is 8,760 hours, or 525,600 minutes. Availability targets are quoted as nines, and the allowed downtime falls fast.

Four nines means your total budget for planned maintenance, failed deployments, provider incidents and human error is under an hour for the whole year. Every extra nine costs roughly an order of magnitude more effort.

The formula

Availability = MTBF / (MTBF + MTTR)

MTBF  Mean Time Between Failures  - how long it runs before breaking
MTTR  Mean Time To Recovery       - how long it takes to become usable again

Example: MTBF = 1000 hours, MTTR = 1 hour
         1000 / 1001 = 0.999001 = 99.9 per cent

Cut MTTR to 6 minutes (0.1 h):
         1000 / 1000.1 = 0.99990 = 99.99 per cent

Read that twice. Availability improved by a factor of ten without making anything fail less often. Recovering faster is usually cheaper than failing less. That is why automated restarts, health checks and rollbacks matter more than heroic hardware.

Combining components

IN SERIES - every part must work (a chain)
  A 99.9%  ->  B 99.9%  ->  C 99.9%
  0.999 x 0.999 x 0.999 = 0.997 = 99.7 per cent
  More dependencies always means LESS available.

IN PARALLEL - any one part is enough (redundancy)
  two independent components, each 99.9 per cent
  failure requires both: 0.001 x 0.001 = 0.000001
  availability = 99.9999 per cent
  Redundancy multiplies the failure probability down.

Two rules follow, and they explain most architecture decisions in this path: every extra dependency lowers availability, and redundancy only helps if the copies fail independently. Two servers in the same rack sharing one power feed are not independent.

Important terminology

Real world example

An online store has a web tier, an application tier and a database, each independently 99.9 per cent available. In series that is 99.7 per cent - about 26 hours of downtime a year, which nobody agreed to.

Two changes fix it without buying better hardware:

• Run two instances of the web and application tiers behind a load balancer with health checks, so a failed instance is removed automatically. Those tiers become effectively far more available.
• Give the database a standby in a second location with automatic failover, cutting its MTTR from an hour of manual work to under two minutes.

The parts still fail at the same rate. The system stopped noticing.

Commands

# How long has this machine been up, and what is the load
uptime

# When did it last boot, and list previous boots
who -b
last reboot | head -5

# Has a service been restarting? Look at its state and restart count
systemctl status YOUR_SERVICE
systemctl show YOUR_SERVICE --property=NRestarts

# Kernel and service messages around a failure, newest last
journalctl -u YOUR_SERVICE --since "-1 hour" --no-pager

# Simple external check: response code and total time, repeated
for i in 1 2 3 4 5; do curl -s -o /dev/null -w "%{http_code} %{time_total}s " https://YOUR_DOMAIN; done

Command options worth knowing

Hands on lab

1. Calculate the monthly downtime budget for 99.9 per cent. Then for 99.99 per cent.
2. A service has MTBF of 500 hours and MTTR of 2 hours. Calculate its availability.
3. Now recalculate with MTTR reduced to 5 minutes. State the improvement.
4. A request path crosses four components, each 99.95 per cent available and independent. Calculate the end to end availability.
5. On a machine you own, run systemctl status against a service and check NRestarts. A non zero value that keeps rising is a crash loop hiding behind an active label.

Expected output

1. 99.9%   -> 43.2 minutes per 30 day month
   99.99%  -> 4.32 minutes per 30 day month

2. 500 / (500 + 2)      = 0.99602 = 99.60 per cent

3. 500 / (500 + 0.0833) = 0.99983 = 99.98 per cent
   Same failure rate. Recovery time did all the work.

4. 0.9995 ^ 4 = 0.998 = 99.80 per cent
   Four "very good" components in series are worse than any one of them.

$ systemctl show nginx --property=NRestarts
NRestarts=0

Common mistakes

• Quoting a nines figure with no measurement behind it. If you are not measuring, you do not know your availability.
• Adding redundancy that shares a dependency. Two instances in the same zone, or two paths through one gateway, fail together.
• Chasing MTBF and ignoring MTTR. Faster recovery is nearly always the cheaper nine.
• Health checks that only prove the process is running. A check that returns 200 while the database is unreachable keeps a broken instance in service.
• Forgetting that dependencies subtract. Every external call you add lowers the ceiling of what you can promise.

Troubleshooting

Security considerations

• Availability is a security property. A denial of service attack is an availability attack, so rate limiting and capacity limits are security controls.
• Health check endpoints must not leak internal detail such as versions, hostnames or connection strings.
• Recovery automation runs with real permissions. Scope those permissions tightly, because anything that can rebuild your system can also destroy it.

Best practices

• Agree a target first, then design to it. Do not design and then claim a number.
• Attack MTTR before MTBF: automated restarts, fast rollback, tested runbooks.
• Remove single points of failure one at a time, starting with the one in the path of every request.
• Make health checks test the dependency the request actually needs.
• Measure availability from where the user is, not from inside your own network.

Interview questions

1. Define availability, reliability and fault tolerance, and give an example that has one but not the others.
2. Write the availability formula and explain why reducing MTTR is usually cheaper than increasing MTBF.
3. Three components at 99.9 per cent in series - what is the end to end availability, and why?
4. How much downtime per month does 99.99 per cent allow?
5. Give an example of redundancy that does not actually improve availability.

Mini assignment

Draw the request path of any application you know, listing every component a request touches. Assign each a plausible availability figure, multiply them, and state the result. Then identify the single change that would most improve the number, and say whether it works by raising MTBF or lowering MTTR.

Conclusion

Availability is time based, reliability is failure based, fault tolerance is about surviving a break while it is happening. Dependencies in series subtract, independent redundancy multiplies the failure away, and cutting recovery time is the cheapest nine you will ever buy.