5. Latency vs Throughput
The picture to keep in your head
Every question below is answered twice: first on the highway, so you can picture it, then in the system, so you know exactly how it works. Each answer ends with a memory code to lock it in.
The picture to keep in your head
Think of a highway between two cities. Two different numbers describe it. How long does one car take to get from A to B? And how many cars pass the toll booth every hour? A sports car makes your trip faster, but it doesn't let more cars through. Adding lanes lets more cars through, but your own trip isn't any faster. And when too many cars arrive at once, everyone sits in a jam, and the trip that took one hour takes three.
Those two numbers are latency and throughput, and most performance confusion comes from mixing them up.
Our running example: the bookstore's book page from Chapters 3 and 4, GET /books/42. To build it, the server reads the book from the database and calls a few other services (price, stock, reviews, recommendations).
| On the highway | In the system |
|---|---|
| Travel time for one car | Latency: how long one request takes |
| Cars past the toll booth per hour | Throughput: requests completed per second |
| Number of lanes | Bandwidth or capacity: the most the system could carry |
| Cars stuck in a jam | Queueing: requests waiting for a free worker |
| The trip stuck behind a slow truck | Tail latency: the slowest few requests |
| A bus that waits to fill up | Batching: more throughput, more waiting |
Figure: the highway, mapped to the system. A faster car doesn't add lanes, and more lanes don't make your car faster.