5. Latency vs Throughput
1
The picture to keep in your head2
5.1 What is latency, exactly?3
5.2 What is throughput, exactly?4
5.3 What's the difference between bandwidth and throughput?5
5.4 Why do averages lie, and what are p50, p95 and p99?6
5.5 What is tail latency, and why does fan-out make it worse?7
5.6 What is Little's law?8
5.7 Where does the time go inside one request?9
5.8 How do you reduce latency?10
5.9 How do you increase throughput?11
5.10 When do latency and throughput fight each other?12
Memory card and where the analogy breaks13
Practice: five levels14
Interview questions15
Solutions16
Sources17
Code lab5. Latency vs Throughput
5.9 How do you increase throughput?
3 min read+10 XP
On the highway: To get more cars through per hour: add lanes, add toll booths (the narrowest point decides everything), and put people on buses so each vehicle carries more.
In the system:
- Find the bottleneck. Throughput is capped by the slowest part. Adding app servers does nothing if the database is the limit. Measure which resource runs out first: CPU, database connections, disk, network, a lock.
- Add capacity there: more app servers (scale out, Chapter 6), read replicas (Chapter 14), more partitions (Chapter 15), more consumers on a queue (Chapter 22).
- Do more at once: more worker threads or async I/O, so a server isn't idle while waiting for the database. Little's law (5.6) tells you how much concurrency you need.
- Batch: write 100 rows in one database call instead of 100 calls; send messages in groups. Each trip carries more.
- Do less per request: caching, smaller responses, cheaper queries.
Remember the bottleneck moves. Fix the database and the next limit might be the network or a single lock in your code. Measure again after every change.
Memory code: "Widen the narrowest point." More lanes anywhere else don't help.