Art of Codeartofcode
Lesson 6 of 17

5. Latency vs Throughput

1
The picture to keep in your head
2
5.1 What is latency, exactly?
3
5.2 What is throughput, exactly?
4
5.3 What's the difference between bandwidth and throughput?
5
5.4 Why do averages lie, and what are p50, p95 and p99?
6
5.5 What is tail latency, and why does fan-out make it worse?
7
5.6 What is Little's law?
8
5.7 Where does the time go inside one request?
9
5.8 How do you reduce latency?
10
5.9 How do you increase throughput?
11
5.10 When do latency and throughput fight each other?
12
Memory card and where the analogy breaks
13
Practice: five levels
14
Interview questions
15
Solutions
16
Sources
17
Code lab
5. Latency vs Throughput

5.5 What is tail latency, and why does fan-out make it worse?

3 min read+10 XP

On the highway: Five friends drive to a wedding in five cars, and lunch starts only when all five arrive. Each car is usually on time but sometimes gets stuck behind a slow truck. The more cars you wait for, the more likely one of them hits a truck, and lunch waits for the slowest car.

In the system: Tail latency is the latency of the slowest requests (p99 and beyond). It matters most when one user request fans out into many calls and must wait for all of them.

The bookstore page, for example, might call price, stock, reviews and many recommendation shards in parallel, then wait for every answer before replying. If each call is slow (say, over 500 ms) 1% of the time, the chance that at least one is slow is:

P(at least one slow) = 1 - (1 - 0.01)^n

1 server: 1% slow; 10 servers: 9.6%; 100 servers: 63%.

Figure: with 100 parallel calls, a "1 in 100" slowness hits nearly two out of three user requests.

This effect is described in Google's paper "The Tail at Scale" (Dean and Barroso, 2013), using exactly this 1% and 100 servers example.

Ways to tame the tail:

  • Fan out less: fewer, bigger calls; avoid calling 100 services for one page.
  • Timeouts with a fallback: if recommendations aren't back in 200 ms, show the page without them.
  • Hedged requests: if a call hasn't answered within, say, the p95 time, send a second copy to another replica and use whichever answers first. The paper describes this; it costs a little extra load.
  • Find the cause: slow disks, garbage-collection pauses, noisy neighbours on shared machines, or one overloaded shard.

Memory code: "Lunch waits for the slowest car." The more calls you wait for, the more the tail becomes the normal case.