5. Latency vs Throughput
1
The picture to keep in your head2
5.1 What is latency, exactly?3
5.2 What is throughput, exactly?4
5.3 What's the difference between bandwidth and throughput?5
5.4 Why do averages lie, and what are p50, p95 and p99?6
5.5 What is tail latency, and why does fan-out make it worse?7
5.6 What is Little's law?8
5.7 Where does the time go inside one request?9
5.8 How do you reduce latency?10
5.9 How do you increase throughput?11
5.10 When do latency and throughput fight each other?12
Memory card and where the analogy breaks13
Practice: five levels14
Interview questions15
Solutions16
Sources17
Code lab5. Latency vs Throughput
5.2 What is throughput, exactly?
3 min read+10 XP
On the highway: Throughput is how many cars pass the toll booth per hour. It says nothing about any one car. A highway can move 6,000 cars an hour even if each trip takes three hours.
In the system: Throughput is the number of requests (or bytes, or jobs) the system completes per unit of time: requests per second, messages per second, MB per second.
- It's about the whole system over time, not one request.
- It's measured on completed work. Requests that were accepted but are still waiting don't count yet.
- Every system has a maximum throughput, set by its slowest part (the bottleneck). Push more requests in than that and they don't go faster; they pile up.
Why both numbers matter: latency is what one user feels. Throughput is how many users you can serve. A system can be fast for one user and fall over at 100, or slow for each user but able to handle millions (think of a batch job that takes an hour but processes a billion rows).
Memory code: "Throughput is cars past the toll booth." A count over time, not one trip.