Art of Codeartofcode
Lesson 14 of 18

29. Failures, Retries, Circuit Breakers

1
The picture to keep in your head
2
29.1 What kinds of failure are there?
3
29.2 What are single points of failure and redundancy?
4
29.3 How does the "nines" math work?
5
29.4 Why do you need timeouts?
6
29.5 How do retries with exponential backoff and jitter work?
7
29.6 What is a retry storm?
8
29.7 What is a circuit breaker?
9
29.8 What are bulkheads?
10
29.9 What is graceful degradation?
11
29.10 How do health checks and failover work?
12
29.11 What is disaster recovery, in brief (RPO, RTO, multi-region)?
13
Memory card and where the analogy breaks
14
Practice: five levels
15
Interview questions
16
Solutions
17
Sources
18
Code lab
29. Failures, Retries, Circuit Breakers

Practice: five levels

3 min read+10 XP

Assignments

Five levels. Do them in order. Answers are in "Solutions" at the end, so try first.

Level 1: Think (no code)

A. Which pattern? One line each.

  1. The payment gateway is down; checkout should stop waiting on it immediately.
  2. Slow recommendations are using up all of checkout's threads.
  3. One request in 200 to inventory fails randomly and succeeds on a second try.
  4. A dependency sometimes hangs for minutes.
  5. Reviews are down; the book page should still load.

B. True or false? Explain in one line.

  1. Two database replicas on the same rack remove the single point of failure.
  2. Retries always improve availability.
  3. A circuit breaker should count 404 responses as failures.
  4. A readiness check should fail if an optional dependency is down.
  5. An untested backup is as good as a tested one.

Level 2: Calculate

  1. How many minutes of downtime per 30-day month does 99.95% allow? Per year for 99.99%?
  2. Checkout needs payment (99.9%), inventory (99.9%) and itself (99.95%). What's checkout's availability?
  3. Two independent load balancers, each 99.5%, either able to do the job. Combined availability?
  4. Three layers each make up to 3 attempts. How many calls can one user action cause at the bottom layer?
  5. A service handles 1,000 requests a second; recommendations take 5 s during an incident and get 100 requests a second. How many threads would they occupy?
  6. Async replication lags 2 s; the store takes 50 orders a second. Roughly how many orders could be lost in a regional failure?

Level 3: Build

In Python:

  1. downtime_minutes(availability, days), serial(*parts) and parallel(a, copies).
  2. percentile(samples, p) and use it to pick a timeout from simulated latencies.
  3. simulate(...) showing a retry storm after a short outage, and recovery with a retry budget.
  4. A CircuitBreaker with closed, open and half-open states.
  5. A bulkhead check: one shared pool vs per-dependency pools when one dependency slows down.

Level 4: Debug

Late one night, real customers couldn't check out for 30 seconds at a time, repeatedly. Logs show a search-engine crawler requesting thousands of non-existent book IDs, which the catalogue answers with 404. Checkout calls the catalogue to confirm prices. The breaker:

python
  1. Explain how a crawler's 404s blocked real customers.
  2. Fix it, and name two other improvements.

Level 5: Design

Make the bookstore's checkout resilient. It calls: payment gateway (critical, external, sometimes slow), inventory (critical, internal), pricing (critical, internal), recommendations (optional), and sends a confirmation email (optional). Peak: 1,000 checkouts a minute during the Big Book Sale. Target: 99.95% availability for checkout.

Deliver:

  1. Timeouts and retries for each dependency, with reasoning.
  2. Where circuit breakers and bulkheads go.
  3. What degrades, and how, when each dependency fails.
  4. How the system avoids retry storms during a payment gateway outage.
  5. Health checks, failover and a DR plan (RPO and RTO) for the order database.