29. Failures, Retries, Circuit Breakers
1
The picture to keep in your head2
29.1 What kinds of failure are there?3
29.2 What are single points of failure and redundancy?4
29.3 How does the "nines" math work?5
29.4 Why do you need timeouts?6
29.5 How do retries with exponential backoff and jitter work?7
29.6 What is a retry storm?8
29.7 What is a circuit breaker?9
29.8 What are bulkheads?10
29.9 What is graceful degradation?11
29.10 How do health checks and failover work?12
29.11 What is disaster recovery, in brief (RPO, RTO, multi-region)?13
Memory card and where the analogy breaks14
Practice: five levels15
Interview questions16
Solutions17
Sources18
Code lab29. Failures, Retries, Circuit Breakers
Memory card and where the analogy breaks
3 min read+10 XP
The memory card (tear this page out)
| # | Question | Memory code |
|---|---|---|
| 29.1 | Kinds of failure? | Crashes are easy; slow, partial and shared failures are the dangerous ones |
| 29.2 | SPOFs and redundancy? | Two pumps, on different circuits |
| 29.3 | Nines math? | Each nine is ten times less downtime; chains lose nines, copies gain them |
| 29.4 | Timeouts? | Don't stand by the kettle all day |
| 29.5 | Retries with backoff? | Wait, then try; wait longer, try again; then stop |
| 29.6 | Retry storms? | When everyone flicks the switch at once, the lights never come back |
| 29.7 | Circuit breakers? | Trip the bad circuit, wait, test once, then trust again |
| 29.8 | Bulkheads? | Separate circuits: the AC can trip without spoiling the milk |
| 29.9 | Graceful degradation? | When power is short, keep the lights on and the AC off |
| 29.10 | Health checks and failover? | Watch the mains; switch over fast, but not at every flicker |
| 29.11 | Disaster recovery? | Decide how much you can lose and how long you can wait, then practise the move |
The whole chapter in one line: Remove single points of failure, put a timeout on every call, retry calmly within a budget, trip breakers on sick dependencies, isolate resources with bulkheads, degrade instead of dying, and practise recovering from the worst.
Where the analogy breaks
- A tripped fuse is obvious. A slow service isn't: it keeps answering, just too late, which is why timeouts matter more than crash detection.
- One house flicking switches doesn't affect the grid much. Thousands of clients retrying together can, which is why retries need budgets and jitter.
- An inverter's job is fixed. Graceful degradation needs product decisions about what matters most, made before the outage.
Quick check
Five questions on this chapter. Pick an answer, then check it.
quiz