29. Failures, Retries, Circuit Breakers
1
The picture to keep in your head2
29.1 What kinds of failure are there?3
29.2 What are single points of failure and redundancy?4
29.3 How does the "nines" math work?5
29.4 Why do you need timeouts?6
29.5 How do retries with exponential backoff and jitter work?7
29.6 What is a retry storm?8
29.7 What is a circuit breaker?9
29.8 What are bulkheads?10
29.9 What is graceful degradation?11
29.10 How do health checks and failover work?12
29.11 What is disaster recovery, in brief (RPO, RTO, multi-region)?13
Memory card and where the analogy breaks14
Practice: five levels15
Interview questions16
Solutions17
Sources18
Code lab29. Failures, Retries, Circuit Breakers
29.3 How does the "nines" math work?
3 min read+10 XP
At home: "The power was out about 9 hours this year" sounds bad. "99.9% uptime" sounds great. They're the same thing.
In the system: Availability is the fraction of time (or of requests) a service works. Each extra nine cuts allowed downtime by 10×:
Figure: every extra nine costs much more; chains lose nines, parallel copies gain them.
| Availability | Down per 30-day month | Down per year |
|---|---|---|
| 99% | 7.2 hours | 3.65 days |
| 99.9% | 43 minutes | 8.8 hours |
| 99.99% | 4.3 minutes | 53 minutes |
| 99.999% | 26 seconds | 5.3 minutes |
Two rules:
- In series (all parts needed), availabilities multiply: checkout needs payment (99.9%), inventory (99.9%) and itself (99.95%): about 99.75%, lower than any single part.
- In parallel (any one is enough), failure probabilities multiply: two independent copies at 99% each give 1 − 0.01 × 0.01 = 99.99%.
So: keep the critical path short, make non-essential calls optional (a down recommendations service shouldn't count against checkout), and add redundancy where it buys real nines. Chapter 30 turns these numbers into targets (SLOs) and error budgets.
Memory code: "Each nine is ten times less downtime; chains lose nines, copies gain them."