5. Latency vs Throughput
1
The picture to keep in your head2
5.1 What is latency, exactly?3
5.2 What is throughput, exactly?4
5.3 What's the difference between bandwidth and throughput?5
5.4 Why do averages lie, and what are p50, p95 and p99?6
5.5 What is tail latency, and why does fan-out make it worse?7
5.6 What is Little's law?8
5.7 Where does the time go inside one request?9
5.8 How do you reduce latency?10
5.9 How do you increase throughput?11
5.10 When do latency and throughput fight each other?12
Memory card and where the analogy breaks13
Practice: five levels14
Interview questions15
Solutions16
Sources17
Code lab5. Latency vs Throughput
Memory card and where the analogy breaks
3 min read+10 XP
The memory card (tear this page out)
| # | Question | Memory code |
|---|---|---|
| 5.1 | Latency? | Latency is one car's trip (and say where the clock starts) |
| 5.2 | Throughput? | Throughput is cars past the toll booth |
| 5.3 | Bandwidth vs throughput? | Lanes are what's possible; the toll count is what happened |
| 5.4 | Percentiles? | The average describes nobody. Ask p50 how it usually goes, and p99 who's suffering. |
| 5.5 | Tail latency and fan-out? | Lunch waits for the slowest car |
| 5.6 | Little's law? | Cars on the road = cars per hour × hours on the road (L = λ × W) |
| 5.7 | Where time goes? | Find the toll booth before you buy a faster car |
| 5.8 | Reducing latency? | Skip it, batch it, move closer, avoid rush hour |
| 5.9 | Increasing throughput? | Widen the narrowest point |
| 5.10 | When they fight? | The bus moves more people, but you wait for the bus |
The whole chapter in one line: Measure the trip and the toll count separately, watch the slowest car, and never run the road full.
Where the analogy breaks
- Cars on a road don't get slower when the road is busier, except in jams. Servers do: they share CPU, memory and locks, so each request does slightly more work when the server is busy.
- A road doesn't cancel your trip. Systems do: timeouts, dropped requests and retries mean some "cars" never arrive, and retries add even more traffic (Chapter 29).
- A car can't be in two places. A request can: hedged requests send copies down two roads on purpose and take whichever arrives first.
Quick check
Five questions on this chapter. Pick an answer, then check it.
quiz