Art of Codeartofcode
Lesson 9 of 17

5. Latency vs Throughput

1
The picture to keep in your head
2
5.1 What is latency, exactly?
3
5.2 What is throughput, exactly?
4
5.3 What's the difference between bandwidth and throughput?
5
5.4 Why do averages lie, and what are p50, p95 and p99?
6
5.5 What is tail latency, and why does fan-out make it worse?
7
5.6 What is Little's law?
8
5.7 Where does the time go inside one request?
9
5.8 How do you reduce latency?
10
5.9 How do you increase throughput?
11
5.10 When do latency and throughput fight each other?
12
Memory card and where the analogy breaks
13
Practice: five levels
14
Interview questions
15
Solutions
16
Sources
17
Code lab
5. Latency vs Throughput

5.8 How do you reduce latency?

3 min read+10 XP

Making it better

On the highway: Four ways to get there sooner. Don't make the trip (have it delivered, or keep one at home). Take fewer trips (buy everything in one go). Live closer. And avoid rush hour, when the road is jammed.

In the system: The same four ideas.

  1. Skip the work: caching (Chapter 12), precomputing results, serving static content from a CDN (Chapter 26).
  2. Fewer round trips, and in parallel: batch several small calls into one; make independent calls at the same time rather than one after another; reuse connections (Chapter 3); add the right database index so one query replaces a scan (Chapter 9).
  3. Move closer: servers or CDN edges near users; keep services that talk a lot in the same data centre.
  4. Keep servers out of rush hour: latency explodes as a server gets busy, because requests start queueing.

latency vs utilisation: flat at first, then shooting up past about 80%.

Figure: a server that needs 10 ms per request answers in 20 ms at 50% busy, 100 ms at 90% and 1,000 ms at 99%.

That last curve comes from basic queueing maths: for a simple server with random arrivals, average time in the system = 1 ÷ (service rate − arrival rate). With 10 ms of work per request (100 per second possible), 90 requests a second gives 1 ÷ (100 − 90) = 0.1 s = 100 ms, ten times the work itself. Real systems differ in detail, but the shape is always the same: flat, then a cliff. That's why teams keep servers well below full (Chapter 2, section 2.11).

Also: move slow work off the request path. Sending the order email, updating recommendations or resizing images can go on a queue and happen after the response (Chapter 22).

Memory code: "Skip it, batch it, move closer, avoid rush hour."