Art of Codeartofcode
Lesson 16 of 17

5. Latency vs Throughput

1
The picture to keep in your head
2
5.1 What is latency, exactly?
3
5.2 What is throughput, exactly?
4
5.3 What's the difference between bandwidth and throughput?
5
5.4 Why do averages lie, and what are p50, p95 and p99?
6
5.5 What is tail latency, and why does fan-out make it worse?
7
5.6 What is Little's law?
8
5.7 Where does the time go inside one request?
9
5.8 How do you reduce latency?
10
5.9 How do you increase throughput?
11
5.10 When do latency and throughput fight each other?
12
Memory card and where the analogy breaks
13
Practice: five levels
14
Interview questions
15
Solutions
16
Sources
17
Code lab
5. Latency vs Throughput

Sources

3 min read+10 XP

Source notes

  • Tail latency, the 1% and 100 servers example, and hedged requests: Jeffrey Dean and Luiz André Barroso, "The Tail at Scale", Communications of the ACM, vol. 56, no. 2, February 2013.
  • Little's law: John D. C. Little, "A Proof for the Queuing Formula: L = λW", Operations Research, 1961.
  • Average time in a single-server queue with random (Poisson) arrivals and random (exponential) service times, 1 ÷ (μ − λ): the standard M/M/1 queue result, confirmed in this chapter by simulation.
  • Nagle's algorithm: RFC 896; TCP_NODELAY disables it on most operating systems.
  • All bookstore numbers (request rates, step timings, the waterfall, the 1 Gbps link) are illustrative assumptions. Arithmetic computed 2026-10-08.
  • Level 3 and Level 4 code: tested 2026-10-08 (code/ch05_latency.py, all checks pass).