Art of Codeartofcode
Lesson 17 of 18

29. Failures, Retries, Circuit Breakers

1
The picture to keep in your head
2
29.1 What kinds of failure are there?
3
29.2 What are single points of failure and redundancy?
4
29.3 How does the "nines" math work?
5
29.4 Why do you need timeouts?
6
29.5 How do retries with exponential backoff and jitter work?
7
29.6 What is a retry storm?
8
29.7 What is a circuit breaker?
9
29.8 What are bulkheads?
10
29.9 What is graceful degradation?
11
29.10 How do health checks and failover work?
12
29.11 What is disaster recovery, in brief (RPO, RTO, multi-region)?
13
Memory card and where the analogy breaks
14
Practice: five levels
15
Interview questions
16
Solutions
17
Sources
18
Code lab
29. Failures, Retries, Circuit Breakers

Sources

3 min read+10 XP

Source notes

  • Circuit breakers and bulkheads: Michael T. Nygard, Release It! (Pragmatic Bookshelf, 2007; 2nd ed. 2018). Martin Fowler, "CircuitBreaker", 2014.
  • Exponential backoff with full jitter and retry budgets: Marc Brooker, "Timeouts, retries, and backoff with jitter", Amazon Builders' Library; AWS Architecture Blog, "Exponential Backoff and Jitter", 2015.
  • Cascading failures and load shedding: Betsy Beyer et al., Site Reliability Engineering (Google, O'Reilly 2016), chapters "Handling Overload" and "Addressing Cascading Failures".
  • Metastable failures: Nathan Bronson et al., "Metastable Failures in Distributed Systems", HotOS 2021.
  • Liveness and readiness probes: Kubernetes documentation.
  • DR strategies (backup and restore, pilot light, warm standby, multi-site active-active): AWS Well-Architected Framework, Reliability Pillar.
  • Availability figures, latency distributions, pool sizes and traffic numbers are illustrative assumptions; arithmetic computed 2026-10-08.
  • Level 3 and Level 4 code: tested 2026-10-08 (code/ch29_resilience.py, all checks pass).