Interview questions

Interview questions (with the counter-questions you'll get)

Q1. What's the difference between latency and throughput?

They're testing: clean definitions and how they relate.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Latency is how long one request takes; throughput is how many requests the system completes per second. They're linked by Little's law: requests in flight equal throughput times latency. You can improve one without the other: more servers raise throughput without making any single request faster, and caching can cut latency. Some techniques, like batching, raise throughput but add latency.
3
4**Counter-questions:**
5- *"Can latency go up when throughput goes up?"* → Yes. As servers get busier, requests queue, so latency rises, slowly at first and then sharply near full utilisation.
6- *"Which matters more?"* → Depends on the path. User-facing requests care about latency; batch jobs and pipelines care about throughput.
7
8**Trap:** treating them as the same thing, or as simple opposites.

Q2. Why use percentiles instead of the average?

They're testing: whether you've debugged real latency.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Latency distributions have long tails. A few very slow requests pull the mean up, so it matches no real request, and the mean also hides how bad the slowest ones are. p50 shows the typical case and p95 or p99 show what the unluckiest users feel, which is usually where problems hide.
3
4**Counter-questions:**
5- *"How do you compute p99 across 50 servers?"* → Merge the raw samples or, more practically, combine histograms, then compute the percentile. Never average per-server percentiles.
6- *"Why care about p99.9?"* → At high volume, 0.1% is still many users, and with fan-out the tail of each part becomes the typical experience of the whole.
7
8**Trap:** "p99 means 99% of users are fine, so it doesn't matter".

Q3. What is tail latency, and how do you reduce it?

They're testing: depth on a known hard problem.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Tail latency is the latency of the slowest requests, like p99. It matters especially when a request fans out to many servers, because the whole request waits for the slowest part: if each of 100 servers is slow 1% of the time, about 63% of requests hit at least one slow server. To reduce it: fan out less, set timeouts with fallbacks, send hedged requests to another replica after a delay, and fix sources of slowness like garbage-collection pauses, overloaded shards and noisy neighbours.
3
4**Counter-questions:**
5- *"What's the cost of hedged requests?"* → Extra load, so you only hedge after waiting around the p95 time, which adds only a few percent more requests.
6- *"Where does the 63% come from?"* → 1 − 0.99^100 ≈ 0.63.
7
8**Trap:** only suggesting "add more servers".

Q4. State Little's law and use it.

They're testing: a practical formula under pressure.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** The average number of requests in a system equals the arrival rate times the average time each spends inside: L = λ × W. At 5,000 requests a second and 200 ms each, about 1,000 requests are in flight, so I need roughly that much concurrency in worker threads, connections and memory. If latency rises to 2 seconds at the same traffic, in-flight requests jump to 10,000, which is how pools get exhausted.
3
4**Counter-questions:**
5- *"A connection pool has 50 connections and queries take 20 ms. Max throughput?"* → 50 ÷ 0.02 = 2,500 queries a second.
6
7**Trap:** knowing the formula but not applying it to sizing.

Q5. Why does latency shoot up when a server gets busy?

They're testing: queueing intuition.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Because requests arrive randomly, not evenly. As utilisation rises, the chance that a new request finds the server busy rises, and the queue grows faster than linearly. For a simple model, average time in the system is 1 ÷ (service rate − arrival rate), so with 10 ms of work it's 20 ms at 50% busy, 100 ms at 90% and 1 second at 99%. That's why we keep headroom.
3
4**Counter-questions:**
5- *"So what utilisation do you target?"* → Often 50 to 70% for latency-sensitive services, higher for batch work, adjusted by measuring the latency curve under load.
6
7**Trap:** "latency rises linearly with load".

Q6. Your API's p99 jumped from 200 ms to 2 seconds, but p50 didn't change. What do you look at?

They're testing: debugging the tail.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Something affects a minority of requests. I'd check whether the slow requests share something: one server or zone, one endpoint, large users or large payloads, a specific downstream call, cache misses, or a time pattern like garbage-collection pauses or a cron job. Traces of slow requests show which step grew. Then I'd look at that component's saturation and errors.
3
4**Counter-questions:**
5- *"All slow requests hit the same database shard."* → A hot shard or a slow query on that shard; look at its load, locks and query plans, and consider rebalancing (Chapter 15).
6
7**Trap:** looking only at averages and calling it fine.

Q7. How would you reduce the latency of a page that calls five services one after another?

They're testing: practical optimisation thinking.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** First measure each call. Then run independent calls in parallel, so the total is the slowest call instead of the sum. Cache results that don't change often, move non-essential work like analytics off the request path onto a queue, set timeouts with fallbacks for optional parts, and reuse connections to each service.
3
4**Counter-questions:**
5- *"Two of the calls depend on each other."* → Run that pair in sequence and everything else in parallel alongside them; or see whether one service can return what the other needs.
6
7**Trap:** jumping to "rewrite in a faster language".

Q8. How would you increase the throughput of a service?

They're testing: bottleneck thinking.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Find the bottleneck first, the resource that saturates: CPU, a database, connections, network, a lock. Then add capacity there: scale out stateless servers, add read replicas or partitions, increase concurrency with async I/O, batch work, or reduce work per request with caching. Then measure again, because the bottleneck moves.
3
4**Counter-questions:**
5- *"You doubled app servers and throughput didn't change. Why?"* → The bottleneck is elsewhere, often the database or a shared dependency.
6
7**Trap:** "add more servers" without finding the bottleneck.

Q9. What's the difference between bandwidth and throughput, and between bandwidth and latency?

They're testing: precise vocabulary.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Bandwidth is the maximum rate a link can carry; throughput is what you actually achieve, always at or below it. Latency is a different dimension: how long data takes to get across. A link can have high bandwidth and high latency, like a fat cable across an ocean.
3
4**Counter-questions:**
5- *"Will a faster internet plan make a far-away website load faster?"* → Only partly. More bandwidth helps big downloads, but each round trip still takes the same time, and many small round trips dominate page loads.
6
7**Trap:** using "bandwidth" and "speed" interchangeably.

Q10. Give an example where improving throughput hurts latency.

They're testing: trade-off awareness.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Batching: waiting to collect 100 writes into one database call raises throughput but makes each write wait for the batch. Nagle's algorithm in TCP does the same for small packets. Running servers at high utilisation also gets more work done per machine but makes queueing delays grow. You cap batch waits with a timer, disable Nagle for latency-sensitive traffic, and keep headroom.
3
4**Counter-questions:**
5- *"And an example where one change helps both?"* → Caching: requests skip work, so each is faster and servers can handle more.
6
7**Trap:** not being able to name a concrete mechanism.
8
9---
10
11## Practice coverage
12
13Every section in this chapter is tested at least once.
14
15| Section | Tested by |
16|---|---|
17| 5.1 Latency | Level 1A, Q1 |
18| 5.2 Throughput | Level 1A, Level 2.2, Q1 |
19| 5.3 Bandwidth vs throughput | Level 2.4, Q9 |
20| 5.4 Percentiles | Level 1B, Level 3, Level 4, Q2 |
21| 5.5 Tail latency and fan-out | Level 2.3, Level 5.3, Q3 |
22| 5.6 Little's law | Level 2.1, Level 3, Q4 |
23| 5.7 Where time goes | Level 5.1, Q6, Q7 |
24| 5.8 Reducing latency | Level 2.2, Level 5.2, Q5, Q7 |
25| 5.9 Increasing throughput | Level 1A.3, Q8 |
26| 5.10 When they fight | Level 5.5, Q10 |
27
28---

5. Latency vs Throughput