Art of Codeartofcode
Lesson 9 of 18

29. Failures, Retries, Circuit Breakers

1
The picture to keep in your head
2
29.1 What kinds of failure are there?
3
29.2 What are single points of failure and redundancy?
4
29.3 How does the "nines" math work?
5
29.4 Why do you need timeouts?
6
29.5 How do retries with exponential backoff and jitter work?
7
29.6 What is a retry storm?
8
29.7 What is a circuit breaker?
9
29.8 What are bulkheads?
10
29.9 What is graceful degradation?
11
29.10 How do health checks and failover work?
12
29.11 What is disaster recovery, in brief (RPO, RTO, multi-region)?
13
Memory card and where the analogy breaks
14
Practice: five levels
15
Interview questions
16
Solutions
17
Sources
18
Code lab
29. Failures, Retries, Circuit Breakers

29.8 What are bulkheads?

3 min read+10 XP

At home: The AC, the fridge and the lights are on separate circuits. When the AC trips, the fridge keeps running. If everything shared one circuit, one faulty appliance would cut the whole house.

In the system: A bulkhead (named after a ship's watertight compartments) isolates resources so one failing dependency can't use up everything:

  • Separate thread or connection pools per dependency.
  • Separate instances for critical and non-critical traffic (checkout servers vs browse servers).
  • Separate queues per customer tier or job type (Chapter 22).
  • Cells: split the whole system into independent copies, each serving a subset of users, so a bad deploy or overload hits one cell, not everyone.

one shared pool is exhausted by slow recommendations; separate pools keep payment and inventory working.

Figure: a slow dependency can only drown its own compartment.

In the chapter's code, recommendations slow to 5 seconds per call. With one shared pool of 200 threads, checkout needs 20 (payment) + 5 (inventory) + 500 (recommendations) = 525 threads: the pool is exhausted and checkout stops. With separate pools (60 for payment, 40 for inventory, 100 for recommendations), only recommendations runs out; its calls fail fast and the page shows bestsellers instead. Checkout keeps working.

Memory code: "Separate circuits: the AC can trip without spoiling the milk."