1Show the Level 1 solution
2**A.**
31. **Circuit breaker** (with a fallback or a clear error).
42. **Bulkhead** (separate pools), plus a timeout.
53. **Retry** with backoff (if the call is idempotent).
64. **Timeout.**
75. **Graceful degradation** (hide the section).
8
9**B.**
101. **False.** One rack failure takes both; spread across racks and zones.
112. **False.** During outages they add load and can cause storms.
123. **False.** 404 is a normal answer, not a sign the service is sick.
134. **False.** It can take every instance out at once; check only essentials.
145. **False.** Untested backups often fail to restore.
1Show the Level 2 solution
21. 99.95%: 0.0005 × 43,200 ≈ **21.6 minutes** a month. 99.99%: 0.0001 × 525,600 ≈ **53 minutes** a year.
32. 0.999 × 0.999 × 0.9995 ≈ **99.75%**.
43. 1 − 0.005 × 0.005 = **99.9975%** (if independent).
54. 3 × 3 × 3 = **27** calls.
65. 100 × 5 = **500** threads (Little's law).
76. 2 × 50 = about **100 orders**.
1Show the Level 3 solution
2Tested solution (Python 3), the key parts:
3
4```python
5def serial(*parts):
6 out = 1.0
7 for a in parts:
8 out *= a
9 return out
10
11
12def parallel(a, copies):
13 return 1 - (1 - a) ** copies
14
15
16def simulate(capacity, base_load, blip_at, blip_len, retry_factor, budget=None,
17 seconds=60, max_attempts=4):
18 retries_next, history = 0.0, []
19 for t in range(seconds):
20 cap = 0 if blip_at <= t < blip_at + blip_len else capacity
21 offered = base_load + retries_next
22 served = min(offered, cap)
23 failed = offered - served
24 retries_next = min(failed * retry_factor, base_load * (max_attempts - 1))
25 if budget is not None:
26 retries_next = min(retries_next, budget * base_load) # retry budget
27 history.append((t, round(offered), round(served)))
28 return history
29
30
31class CircuitBreaker:
32 def __init__(self, threshold=5, cooldown=30, counts=lambda status: status >= 500):
33 self.threshold, self.cooldown, self.counts = threshold, cooldown, counts
34 self.state, self.failures, self.opened_at = "closed", 0, None
35
36 def allow(self, now):
37 if self.state == "open" and now - self.opened_at >= self.cooldown:
38 self.state = "half-open"
39 return self.state != "open"
40
41 def record(self, status, now):
42 if self.counts(status):
43 self.failures += 1
44 if self.state == "half-open" or self.failures >= self.threshold:
45 self.state, self.opened_at = "open", now
46 else:
47 if self.state == "half-open":
48 self.state = "closed"
49 self.failures = 0
50```
51
52All checks pass: the nines table matches; checkout's chain is 99.75% and two 99% copies give 99.99%; simulated latency has p50 ≈ 89 ms, p99 ≈ 231 ms, p99.9 ≈ 312 ms; a 3-second outage with naive retries stays stuck at 3,200 offered requests a second while a 10% budget recovers in a second; the breaker opens, refuses, half-opens and closes; one shared pool stalls checkout while bulkheads isolate recommendations; and the 404-counting breaker blocks real checkouts. The full file is in `code/ch29_resilience.py`.
1Show the Level 4 solution
21. **What happened:** the breaker counts **every** status ≥ 400 as a failure, including **404**. The crawler's requests for non-existent books produce a stream of 404s from the catalogue; after 5 in a row, the breaker **opens**, and for the next 30 seconds **all** catalogue calls fail fast, including checkout's price checks for real customers. Then it closes, the crawler sends more 404s, and it opens again: the repeated 30-second outages.
32. **The fix:** count only failures that mean the dependency is unhealthy:
4
5```python
6def is_failure(status):
7 return status >= 500 or status in (408, 429) or status == "timeout"
8
9class CircuitBreaker:
10 def record(self, status, now):
11 if is_failure(status):
12 ...
13 else:
14 ... # 2xx, 3xx and ordinary 4xx are healthy answers
15```
16
17**Also:**
18- Use a **failure rate over a window** with a minimum number of calls (for example, 50% of at least 20 calls in 10 seconds), not "5 in a row".
19- **Separate breakers per endpoint or caller**: checkout's price lookups shouldn't share a breaker with public catalogue browsing.
20- Rate-limit or block the crawler at the edge (Chapter 25), and cache 404s briefly.
1Show the Level 5 (sketch) solution
21. **Timeouts and retries:**
3 - Payment gateway: connect 200 ms, read 3 s (external, sometimes slow); retry once on timeout or 503, **with an idempotency key** (Chapter 24), full jitter.
4 - Inventory and pricing: 300 ms timeouts from their p99.9; one retry with jitter.
5 - Recommendations: 150 ms, no retry.
6 - Email: sent asynchronously via a queue (Chapter 22); no synchronous call at all.
7 - Overall checkout deadline of 5 s, propagated downstream.
82. **Breakers and bulkheads:** a breaker per dependency (failure rate over a window, counting timeouts and 5xx only); separate connection pools: payment 100, inventory 50, pricing 50, recommendations 30. Checkout runs on separate instances from browsing.
93. **Degradation:**
10 - Recommendations down: cached or static bestsellers.
11 - Pricing slow: use the price shown in the cart if it's under 5 minutes old, then re-verify asynchronously.
12 - Inventory down: accept the order as "pending stock confirmation" for in-stock-recently items, or show a clear "try again" message.
13 - Payment gateway A failing: route to gateway B, or show alternative payment methods.
144. **No retry storms:** one retrying layer (checkout service only, not the app plus the gateway client), retry budget of 10% per dependency, breakers open on high failure rates, the gateway client uses a token bucket for outbound calls (Chapter 25), and checkout sheds load with a waiting room during sales (Chapter 43).
155. **Health, failover, DR:** readiness checks only the instance's own DB pool; liveness only process health; PostgreSQL primary with a synchronous standby in another zone (RPO ≈ 0 within the region, automatic failover in under a minute), plus asynchronous replication to a second region (RPO of seconds, RTO under 30 minutes with a warm standby). Monthly restore tests and quarterly failover drills.