29.1 What kinds of failure are there?

Expecting failure

At home: A bulb blows (one part dies). A wire gets loose and the light flickers (on and off). The voltage drops and everything runs slowly. A neighbour's short circuit trips your switch. The whole colony loses power.

In the system: Failures come in many shapes, and the nasty ones aren't the clean crashes:

• Crash (fail-stop): a server dies. Easy to detect: it stops answering.
• Slow (gray failure): it answers, but takes 10 seconds instead of 100 ms. Often worse than a crash, because callers wait and pile up (Chapter 27's Level 4).
• Partial: some requests fail (one bad disk, one bad shard), others succeed.
• Intermittent: fails now and then, then recovers; hard to reproduce.
• Network: packets lost, connections reset, a partition (Chapter 16).
• Dependency: your code is fine, but the payment gateway, DNS (Chapter 4) or the CDN (Chapter 26) is down.
• Overload: too much traffic for the capacity.
• Human and deploy errors: a bad config or release. A large share of real outages start with a change.
• Correlated: many things fail together: a whole zone, a shared library bug, a certificate expiring everywhere at once.

The key mindset: at scale, something is always failing. Design so that a failure in one part becomes a smaller problem in that part, not a bigger problem everywhere.

Memory code: "Crashes are easy; slow, partial and shared failures are the dangerous ones."

---