5. Latency vs Throughput
1
The picture to keep in your head2
5.1 What is latency, exactly?3
5.2 What is throughput, exactly?4
5.3 What's the difference between bandwidth and throughput?5
5.4 Why do averages lie, and what are p50, p95 and p99?6
5.5 What is tail latency, and why does fan-out make it worse?7
5.6 What is Little's law?8
5.7 Where does the time go inside one request?9
5.8 How do you reduce latency?10
5.9 How do you increase throughput?11
5.10 When do latency and throughput fight each other?12
Memory card and where the analogy breaks13
Practice: five levels14
Interview questions15
Solutions16
Sources17
Code lab5. Latency vs Throughput
5.1 What is latency, exactly?
3 min read+10 XP
The definitions
On the highway: Latency is your trip: the time from leaving home to arriving. It's about one car. And it depends on where you start the clock: the drive alone, or including the time you waited at the toll booth?
In the system: Latency is the time one request takes, from being sent to the answer arriving. It's measured in milliseconds (ms) or microseconds.
Where you measure changes the number, so always say which:
- Client-side latency (what the user feels): everything, including DNS, handshakes and the network (Chapter 3).
- Server-side latency (what your server logs show): from the request arriving at your server to the response leaving it. It hides the network.
- Service time vs waiting time: inside the server, a request may wait in a queue before any work starts. Latency = waiting + working.
Response time is often used to mean the same thing as latency; in this book they're the same.
Memory code: "Latency is one car's trip." And say where the clock starts.