Art of Codeartofcode
Lesson 8 of 17

5. Latency vs Throughput

1
The picture to keep in your head
2
5.1 What is latency, exactly?
3
5.2 What is throughput, exactly?
4
5.3 What's the difference between bandwidth and throughput?
5
5.4 Why do averages lie, and what are p50, p95 and p99?
6
5.5 What is tail latency, and why does fan-out make it worse?
7
5.6 What is Little's law?
8
5.7 Where does the time go inside one request?
9
5.8 How do you reduce latency?
10
5.9 How do you increase throughput?
11
5.10 When do latency and throughput fight each other?
12
Memory card and where the analogy breaks
13
Practice: five levels
14
Interview questions
15
Solutions
16
Sources
17
Code lab
5. Latency vs Throughput

5.7 Where does the time go inside one request?

3 min read+10 XP

On the highway: A 3-hour trip might be 20 minutes getting out of the city, 2 hours of open road, and 40 minutes stuck at one toll booth. To get there faster, you find the biggest piece first. Buying a faster car does nothing about the toll booth.

In the system: Break one request into its parts before optimising anything. A waterfall shows each step on a timeline.

a waterfall of one first visit to the book page: DNS, TCP, TLS, network, queue, cache, database, build, response.

Figure: in this example, 160 of the 206 ms are network and handshakes. The database query is only 30 ms.

The usual parts:

  1. Network and handshakes: DNS, TCP, TLS, then the request and response travelling (Chapters 3 and 4). Often the biggest part for users far away.
  2. Queueing: waiting for a free worker on the server (5.8).
  3. Work: your code, cache lookups, database queries, calls to other services.
  4. Response size: big responses take longer to send, and longer for the browser to process.

How you find out in real systems: distributed tracing records each step of a request across services (Chapter 30), and the browser's developer tools show the network part.

The lesson: don't guess. In the example above, making the database twice as fast saves 15 ms, while reusing the connection (no TCP or TLS handshakes) saves 80 ms.

Memory code: "Find the toll booth before you buy a faster car." Measure where the time goes, then fix the biggest piece.