Art of Codeartofcode
Lesson 8 of 18

29. Failures, Retries, Circuit Breakers

1
The picture to keep in your head
2
29.1 What kinds of failure are there?
3
29.2 What are single points of failure and redundancy?
4
29.3 How does the "nines" math work?
5
29.4 Why do you need timeouts?
6
29.5 How do retries with exponential backoff and jitter work?
7
29.6 What is a retry storm?
8
29.7 What is a circuit breaker?
9
29.8 What are bulkheads?
10
29.9 What is graceful degradation?
11
29.10 How do health checks and failover work?
12
29.11 What is disaster recovery, in brief (RPO, RTO, multi-region)?
13
Memory card and where the analogy breaks
14
Practice: five levels
15
Interview questions
16
Solutions
17
Sources
18
Code lab
29. Failures, Retries, Circuit Breakers

29.7 What is a circuit breaker?

3 min read+10 XP

At home: The trip switch notices the AC circuit is faulty and cuts it off, instantly and every time, instead of letting it keep sparking. After a while you try it once; if it holds, you leave it on.

In the system: A circuit breaker wraps calls to a dependency and tracks failures (popularised by Michael Nygard's Release It!):

closed until 5 failures, then open; after a cooldown, half-open lets a trial through; success closes, failure reopens.

Figure: fail fast while the dependency is sick, and test carefully before trusting it again.

  • Closed (normal): calls go through; failures are counted.
  • Open: after too many failures (say 5 in a row, or 50% in the last 10 seconds), calls fail immediately without touching the dependency. The caller uses a fallback (29.9).
  • Half-open: after a cooldown (say 30 s), let one (or a few) trial calls through. Success closes the breaker; failure opens it again.

In the chapter's code, five 503s open the breaker; calls are refused at second 10; at second 35 one trial is allowed, succeeds, and the breaker closes.

Why it helps:

  • The caller stays fast: no threads stuck waiting on a dead service.
  • The dependency gets room to recover instead of being hammered.
  • Clear signal for dashboards and alerts: "breaker to payment gateway is open".

Getting it right:

  • Count the right failures: timeouts and 5xx mean "unhealthy"; 4xx don't (a 404 for a missing book is a normal answer). The Level 4 bug counts 404s.
  • One breaker per dependency (or per endpoint), not one for everything.
  • Libraries: resilience4j and Polly; service meshes (Istio, Linkerd) and Envoy offer outlier detection that ejects failing hosts.

Memory code: "Trip the bad circuit, wait, test once, then trust again."