Art of Codeartofcode
Lesson 3 of 18

29. Failures, Retries, Circuit Breakers

1
The picture to keep in your head
2
29.1 What kinds of failure are there?
3
29.2 What are single points of failure and redundancy?
4
29.3 How does the "nines" math work?
5
29.4 Why do you need timeouts?
6
29.5 How do retries with exponential backoff and jitter work?
7
29.6 What is a retry storm?
8
29.7 What is a circuit breaker?
9
29.8 What are bulkheads?
10
29.9 What is graceful degradation?
11
29.10 How do health checks and failover work?
12
29.11 What is disaster recovery, in brief (RPO, RTO, multi-region)?
13
Memory card and where the analogy breaks
14
Practice: five levels
15
Interview questions
16
Solutions
17
Sources
18
Code lab
29. Failures, Retries, Circuit Breakers

29.2 What are single points of failure and redundancy?

3 min read+10 XP

At home: If the house has one water pump and it breaks, no water. Two pumps, either able to do the job, and a broken one is an inconvenience, not a crisis.

In the system: A single point of failure (SPOF) is any component whose failure takes the whole system down. Redundancy removes it with spare capacity that can take over:

  • Stateless services: run several instances behind a load balancer (Chapters 6 and 7) across availability zones.
  • Databases: replicas with automatic failover (Chapter 14).
  • Load balancers, DNS, caches, queues: in pairs or clusters.
  • Regions: for the biggest failures (29.11).

How to find SPOFs: draw the request path and ask of every box, "if this dies, what happens?" Common hidden ones: a single NAT gateway, one DNS provider, one secrets server, a cron job on one machine, one engineer who knows how to deploy.

Redundancy only helps if failures are independent. Two copies on the same rack, the same power supply or the same buggy software version can fail together. Spread copies across zones, and roll out changes gradually so a bad release doesn't hit every copy at once.

Memory code: "Two pumps, on different circuits."