Art of Codeartofcode
Lesson 13 of 17

5. Latency vs Throughput

1
The picture to keep in your head
2
5.1 What is latency, exactly?
3
5.2 What is throughput, exactly?
4
5.3 What's the difference between bandwidth and throughput?
5
5.4 Why do averages lie, and what are p50, p95 and p99?
6
5.5 What is tail latency, and why does fan-out make it worse?
7
5.6 What is Little's law?
8
5.7 Where does the time go inside one request?
9
5.8 How do you reduce latency?
10
5.9 How do you increase throughput?
11
5.10 When do latency and throughput fight each other?
12
Memory card and where the analogy breaks
13
Practice: five levels
14
Interview questions
15
Solutions
16
Sources
17
Code lab
5. Latency vs Throughput

Practice: five levels

3 min read+10 XP

Assignments

Five levels. Do them in order. Answers are in "Solutions" at the end, so try first.

Level 1: Think (no code)

A. Latency problem or throughput problem? Explain in one line.

  1. The search page takes 4 seconds for every user, even at night when nobody's using the site.
  2. At 9 pm, the order API starts returning errors and timeouts; at 3 am it's fine.
  3. A nightly job must process 50 million rows before 6 am and is finishing at 9 am.
  4. A user in Brazil says the site feels slow; users in the same region as your servers don't.

B. Your monitoring shows: mean 90 ms, p50 40 ms, p99 2,100 ms. Your manager says "90 ms average is great, nothing to fix." What do you say?

Level 2: Calculate

  1. The bookstore gets 2,000 requests a second, each taking 50 ms. How many requests are in progress at any moment?
  2. One worker needs 10 ms of work per request. What's its maximum throughput? If 90 requests a second arrive, roughly what's the average latency (use 1 ÷ (service rate − arrival rate))?
  3. The book page calls 20 recommendation shards in parallel and waits for all. Each is slow 1% of the time. What fraction of page loads are slow? What if each shard were slow only 0.1% of the time?
  4. A 1 Gbps link serves 2 MB pages. What's the most pages a second it can deliver?

Level 3: Build

Write Python functions (standard library only):

  1. percentile(samples, p) using the nearest-rank method, and summarize(samples) returning mean, p50, p95, p99 and max.
  2. littles_law(throughput, latency).
  3. fanout_slow_probability(p_slow, n).
  4. simulate_queue(arrival_rate, service_rate): one server, random arrivals and random service times; return the average time in the system.

Check that:

  • For [10]*90 + [50]*9 + [2000], p50 = 10, p95 = 50, p99 = 50, mean = 33.5.
  • littles_law(5000, 0.2) == 1000.
  • fanout_slow_probability(0.01, 100) ≈ 0.634.
  • The simulation matches 1 ÷ (service rate − arrival rate) within 15% at 50% and 90% busy.

Level 4: Debug

The dashboard shows the bookstore's overall p99 latency. It's computed like this from 10 servers:

python

The dashboard says p99 is about 170 ms. Customer complaints say pages often take over a second.

  1. Find the bug. Explain why the number is wrong and in which direction.
  2. Fix it.
  3. Nine servers have p99 ≈ 60 ms and one has p99 ≈ 1,180 ms (30% of its requests take about a second). Roughly what is the real fleet p99, and why?

Level 5: Design

The bookstore's book page must have p99 under 300 ms. Today it's 1.2 seconds. It's built like this, one step after another:

1. read book from the database ~20 ms 2. call price service ~30 ms 3. call stock service ~30 ms 4. call reviews service ~40 ms 5. call 20 recommendation shards ~25 ms each, in parallel, p99 900 ms 6. log the page view to analytics ~50 ms (waits for the answer)

Deliver:

  1. Where the time goes, and the two biggest problems.
  2. A redesigned flow (what runs in parallel, what moves off the request path, what gets cached).
  3. How you stop recommendations from blowing the p99.
  4. What you'd measure to prove it worked.
  5. One trade-off your changes introduce.