1Show the Level 1 solution
2**A.**
31. **Latency.** Slow even with no load, so it's the work per request (a slow query, too many sequential calls, a far-away dependency), not capacity.
42. **Throughput** (capacity). Fine when quiet, failing at peak: the system can't complete requests as fast as they arrive, so they queue and time out.
53. **Throughput.** Nobody waits on one row; the job needs more rows per hour (parallelism, batching, fixing the bottleneck).
64. **Latency**, from distance. Each round trip to far-away servers is long; a CDN, a nearby region or fewer round trips would help.
78**B.** "The average hides the problem. Half our requests take under 40 ms, but 1 in 100 takes over 2 seconds. At our volume that's a lot of users every day, and fan-out pages hit the tail even more often. We should find out what's making the slow 1% slow."
Show the Level 2 solution
1. Little's law: 2,000 × 0.05 = **100 requests in progress**.
2. Max throughput = 1 ÷ 0.01 s = **100 requests a second**. At 90 a second: 1 ÷ (100 − 90) = 0.1 s = **100 ms** average, ten times the 10 ms of actual work.
3. 1 − 0.99²⁰ ≈ **18%** of page loads are slow. At 0.1% each: 1 − 0.999²⁰ ≈ **2%**. Improving each shard's tail by 10 times improves the page's tail by about 9 times.
4. 1,000,000,000 bits ÷ 8 = 125,000,000 bytes a second ÷ 2,000,000 bytes = **62.5 pages a second**.
1Show the Level 3 solution
2Tested solution (Python 3):
34```python
5import math
6import random
789def percentile(samples, p):
10 ordered = sorted(samples)
11 rank = max(1, math.ceil(p / 100 * len(ordered)))
12 return ordered[rank - 1]
131415def summarize(samples):
16 return {"mean": sum(samples) / len(samples), "p50": percentile(samples, 50),
17 "p95": percentile(samples, 95), "p99": percentile(samples, 99), "max": max(samples)}
181920def littles_law(throughput_per_s, latency_s):
21 return throughput_per_s * latency_s
222324def fanout_slow_probability(p_slow, n):
25 return 1 - (1 - p_slow) ** n
262728def simulate_queue(arrival_rate, service_rate, n=200_000, seed=1):
29 rng = random.Random(seed)
30 t = server_free_at = total = 0.0
31 for _ in range(n):
32 t += rng.expovariate(arrival_rate) # next arrival
33 start = max(t, server_free_at) # wait if the server is busy
34 server_free_at = start + rng.expovariate(service_rate)
35 total += server_free_at - t # waiting + service
36 return total / n
37```
3839All checks pass. With 10 ms of work per request, the simulation gives about **20 ms** at 50% busy and about **99 ms** at 90% busy, matching the formula. The full file is in `code/ch05_latency.py`.
1Show the Level 4 solution
2**The bug:** it **averages percentiles**. A percentile of the whole fleet can't be computed from per-server percentiles by averaging.
341. Each server's p99 describes only its own requests. Averaging gives every server equal weight and blends one terrible number with nine good ones: (9 × 60 + 1,180) ÷ 10 ≈ **172 ms**. The real fleet p99 is much higher, so the dashboard **under-reports** the problem.
52. **Fix:** compute the percentile over **all** requests together (or, at scale, merge per-server histograms and compute from the merged histogram):
67```python
8def overall_p99(servers):
9 merged = [x for s in servers for x in s]
10 return percentile(merged, 99)
11```
12133. Server 9 handles 10% of requests and 30% of those are slow, so about **3% of all requests** take around a second. The fleet p99 (the slowest 1%) therefore lands inside that slow group: about **1,065 ms** in the tested simulation, roughly six times what the dashboard showed. The real fix is the slow disk on server 9.
1Show the Level 5 (sketch) solution
21. **Where it goes:** steps 1 to 4 run one after another (about 120 ms); recommendations have a p99 of 900 ms and, with 20 shards, the page hits a slow shard about 18% of the time; analytics adds 50 ms of pure waiting. **Biggest problems:** the recommendation tail and doing everything in sequence.
32. **Redesign:** read the book (20 ms), then call price, stock and reviews **in parallel** (about 40 ms, the slowest of the three). **Cache** reviews and the book data (Chapter 12). Send the page view to analytics **through a queue**, without waiting (Chapter 22). The core page now takes about 60 ms. Recommendations start at the same time as the other calls, capped at 150 ms (point 3), so even the worst case stays around 200 ms, under the 300 ms target.
43. **Recommendations:** give them a **budget** (say 150 ms) and render the page without them (or with a cached generic list) if they're late; use **hedged requests** to another replica after about the shard's p95; precompute recommendations per book overnight so most requests are a cache read; and fix the slow shards.
54. **Measure:** page p50, p95 and p99 from real users and on the server, per-step timings from tracing, the rate of recommendation timeouts and fallbacks, and analytics queue lag.
65. **Trade-offs:** some users see a page **without** fresh recommendations; cached data can be slightly **stale**; analytics arrives seconds late; hedged requests add a few percent **extra load** on recommendation shards.