29. Failures, Retries, Circuit Breakers
1
The picture to keep in your head2
29.1 What kinds of failure are there?3
29.2 What are single points of failure and redundancy?4
29.3 How does the "nines" math work?5
29.4 Why do you need timeouts?6
29.5 How do retries with exponential backoff and jitter work?7
29.6 What is a retry storm?8
29.7 What is a circuit breaker?9
29.8 What are bulkheads?10
29.9 What is graceful degradation?11
29.10 How do health checks and failover work?12
29.11 What is disaster recovery, in brief (RPO, RTO, multi-region)?13
Memory card and where the analogy breaks14
Practice: five levels15
Interview questions16
Solutions17
Sources18
Code lab29. Failures, Retries, Circuit Breakers
29.9 What is graceful degradation?
3 min read+10 XP
Surviving the worst
At home: During a power cut, the inverter runs the lights and fans, not the AC and the geyser. Life goes on, a bit less comfortably.
In the system: When a part fails, keep the core working with less, instead of failing completely. Decide in advance what's essential:
- Recommendations down: show yesterday's cached picks, then a static bestseller list. In the chapter's code, the box falls back in exactly that order.
- Search slow: show cached results for popular queries, or a simpler search (Chapter 19).
- Reviews service down: hide the reviews section; the book page still sells the book.
- Payment gateway A failing: switch to gateway B, or offer other payment methods.
- Overloaded during a sale: turn off expensive extras (personalisation, "people also viewed"), serve more from cache, put a waiting room in front of checkout (Chapter 43).
Tools:
- Fallbacks in code next to each non-critical call, triggered by timeouts or open breakers.
- Feature flags (kill switches) to turn off expensive features instantly.
- Stale data over no data (
stale-if-error, Chapter 26; stale cache entries, Chapter 13). - Load shedding by priority: drop analytics before checkout.
Test the degraded modes. A fallback that has never run usually doesn't work when you need it.
Memory code: "When power is short, keep the lights on and the AC off."