5. Latency vs Throughput
1
The picture to keep in your head2
5.1 What is latency, exactly?3
5.2 What is throughput, exactly?4
5.3 What's the difference between bandwidth and throughput?5
5.4 Why do averages lie, and what are p50, p95 and p99?6
5.5 What is tail latency, and why does fan-out make it worse?7
5.6 What is Little's law?8
5.7 Where does the time go inside one request?9
5.8 How do you reduce latency?10
5.9 How do you increase throughput?11
5.10 When do latency and throughput fight each other?12
Memory card and where the analogy breaks13
Practice: five levels14
Interview questions15
Solutions16
Sources17
Code lab5. Latency vs Throughput
5.10 When do latency and throughput fight each other?
3 min read+10 XP
On the highway: A bus carries 50 people in one vehicle, so many more people cross the city per hour. But you might wait 15 minutes at the stop for the bus to come and fill up. Great for the city's throughput, worse for your trip.
In the system: Several throughput tricks cost latency:
- Batching: collecting writes into a batch before sending means each item waits for the batch to fill (or for a timer). Systems usually cap the wait: "send when 100 items are ready or 10 ms have passed".
- Running servers very busy: high utilisation gets more work out of each machine, but queueing makes each request slower (5.8).
- Small-packet merging: TCP's Nagle's algorithm waits briefly to combine tiny writes into fewer packets. It saves network overhead but adds delay, so latency-sensitive apps (games, trading, chatty protocols) often turn it off with
TCP_NODELAY. - Compression: smaller responses (better for bandwidth) cost CPU time to compress and decompress.
And sometimes they help each other: caching makes requests faster and frees servers to handle more.
The design question: what does this feature need? A checkout button needs low latency. A nightly report or a log pipeline needs high throughput and doesn't mind waiting. Choose per path.
Memory code: "The bus moves more people, but you wait for the bus."