Art of Codeartofcode
Lesson 5 of 17

5. Latency vs Throughput

1
The picture to keep in your head
2
5.1 What is latency, exactly?
3
5.2 What is throughput, exactly?
4
5.3 What's the difference between bandwidth and throughput?
5
5.4 Why do averages lie, and what are p50, p95 and p99?
6
5.5 What is tail latency, and why does fan-out make it worse?
7
5.6 What is Little's law?
8
5.7 Where does the time go inside one request?
9
5.8 How do you reduce latency?
10
5.9 How do you increase throughput?
11
5.10 When do latency and throughput fight each other?
12
Memory card and where the analogy breaks
13
Practice: five levels
14
Interview questions
15
Solutions
16
Sources
17
Code lab
5. Latency vs Throughput

5.4 Why do averages lie, and what are p50, p95 and p99?

3 min read+10 XP

Measuring honestly

On the highway: Most days your commute takes 30 minutes. Once a week a truck breaks down and it takes 2 hours. The "average" says 43 minutes, but you never actually had a 43-minute trip. And the 2-hour day is the one you complain about.

In the system: Request latencies aren't spread evenly. Most requests are fast and a few are very slow, so the mean gets pulled up by the slow ones and describes nobody.

a latency histogram: most requests are fast, a long thin tail of slow ones, with p50, mean, p95 and p99 marked.

Figure: the mean sits where almost no request actually landed. Percentiles tell you what real users experienced.

Percentiles fix this. Sort all the latencies, then:

  • p50 (the median): half of requests were faster than this. The "typical" request.
  • p95: 95% were faster; 1 in 20 was slower.
  • p99: 99% were faster; 1 in 100 was slower.
  • p99.9: 1 in 1,000 was slower.

Why p99 matters more than it sounds: a user loading one page may make dozens of requests, and a busy site serves millions of requests a day. If p99 is 2 seconds, then 1% of requests, possibly tens of thousands a day, take 2 seconds or more. Often the slowest requests belong to your most valuable users: the ones with the biggest carts and the longest order history.

Two rules when you use percentiles:

  • Always say the window: "p99 over the last 5 minutes".
  • Never average percentiles. The p99 of the whole fleet is not the average of each server's p99 (the Level 4 bug in this chapter shows why). Combine the raw measurements, or histograms, then take the percentile.

Memory code: "The average describes nobody. Ask p50 how it usually goes, and p99 who's suffering."