Interview questions

Interview questions (with the counter-questions you'll get)

Q1. How do you make a service resilient to dependency failures?

They're testing: the toolbox, in order.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Timeouts on every call; retries only for transient errors on idempotent operations, with exponential backoff, jitter and a budget; circuit breakers per dependency to fail fast; bulkheads so one slow dependency can't exhaust shared resources; fallbacks and graceful degradation for non-critical features; and async processing for work that can wait.
3
4**Counter-questions:**
5- *"Which first?"* → Timeouts: without them, nothing else helps because callers hang.
6
7**Trap:** "just add retries".

Q2. Explain the circuit breaker pattern.

They're testing: a classic pattern in detail.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** A wrapper that counts failures to a dependency. Closed: calls pass. After a threshold of failures, open: calls fail immediately so the caller stays fast and the dependency can recover. After a cooldown, half-open: allow a trial call; success closes, failure reopens. Count only health failures (timeouts, 5xx), one breaker per dependency.
3
4**Counter-questions:**
5- *"What does the caller do while it's open?"* → A fallback: cached data, a default, a degraded response or a clear error.
6
7**Trap:** no half-open state.

Q3. What's a retry storm, and how do you prevent it?

They're testing: systemic thinking.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** When a service struggles, callers' retries multiply its load, keeping it overloaded even after the original cause ends. Prevent with retry budgets, exponential backoff with jitter, retrying at a single layer, circuit breakers, server-side load shedding, and deadlines so stale requests are dropped.
3
4**Counter-questions:**
5- *"What's a metastable failure?"* → A state where the system stays overloaded through feedback (like retries) even after the trigger is gone.
6
7**Trap:** more retries "for reliability".

Q4. How do you choose a timeout?

They're testing: practical judgement.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** From the dependency's measured latency (a bit above p99 or p99.9), within the caller's own end-to-end budget, with separate connect and read timeouts, and deadlines propagated downstream. Revisit as latency changes.
3
4**Counter-questions:**
5- *"And for a call that's sometimes legitimately slow?"* → Make it async, or use a longer timeout on a separate pool so it can't starve fast calls.
6
7**Trap:** library defaults.

Q5. What is a bulkhead?

They're testing: isolation.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Partitioning resources so a failure in one area can't consume everything: separate thread or connection pools per dependency, separate instances for critical traffic, separate queues per tenant, or full cells. A slow dependency then exhausts only its own pool.
3
4**Counter-questions:**
5- *"Cost?"* → Less resource sharing, so some idle capacity; sizing each pool needs care.
6
7**Trap:** confusing it with a circuit breaker.

Q6. What does 99.99% availability mean in practice?

They're testing: numeracy.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** About 4.3 minutes of downtime a month, or 53 minutes a year. Chained dependencies multiply, so a 99.99% service can't depend synchronously on several 99.9% services; it needs redundancy, caching, async paths and fast detection and recovery.
3
4**Counter-questions:**
5- *"How do you get from 99.9% to 99.99%?"* → Remove SPOFs, multi-zone everything, automate failover, safer deploys with canaries and fast rollback, and degrade instead of failing.
6
7**Trap:** promising five nines without discussing cost.

Q7. Liveness vs readiness checks?

They're testing: operational detail.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Liveness asks whether the process is alive; failing it restarts the instance. Readiness asks whether it can serve traffic now; failing it removes the instance from the load balancer. Readiness should check essentials for this instance, not optional or shared dependencies, or all instances can go unready together.
3
4**Counter-questions:**
5- *"What if the database is down?"* → Failing readiness everywhere takes the whole service offline; often better to stay ready and return fast, clear errors or degraded responses.
6
7**Trap:** deep health checks that cascade.

Q8. What are RPO and RTO?

They're testing: DR vocabulary.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** RPO is how much data loss is acceptable, measured as time since the last recoverable copy; RTO is how long recovery may take. Strategies range from backup-and-restore (hours) through pilot light and warm standby to multi-region active-active (near zero), with rising cost.
3
4**Counter-questions:**
5- *"How do you know you meet them?"* → Regular restore tests and failover drills, timed.
6
7**Trap:** mixing up the two.

Q9. What is graceful degradation? Give examples.

They're testing: product-aware engineering.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Keeping core functionality when parts fail, with reduced features: cached or static recommendations when the service is down, hiding reviews, simpler search, switching payment providers, turning off expensive features with flags during overload, serving stale content.
3
4**Counter-questions:**
5- *"How do you know fallbacks work?"* → Test them regularly, including in production with chaos experiments.
6
7**Trap:** fallbacks that call the same failing dependency.

Q10. Design for a region-wide outage.

They're testing: big-picture resilience.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Decide RPO and RTO per data tier. Replicate data to a second region (async for low cost, sync or consensus for zero loss at latency cost). Run warm standby or active-active compute, route users with DNS or global load balancing based on health, keep configuration and deploys region-independent, and rehearse failover regularly.
3
4**Counter-questions:**
5- *"What breaks in active-active?"* → Conflicting writes across regions; partition data by home region or use conflict resolution.
6
7**Trap:** "we use multiple availability zones" as the whole answer.
8
9---
10
11## Practice coverage
12
13Every section in this chapter is tested at least once.
14
15| Section | Tested by |
16|---|---|
17| 29.1 Kinds of failure | Level 1A.4, Q1 |
18| 29.2 SPOFs and redundancy | Level 1B.1, Level 2.3, Q6 |
19| 29.3 Nines math | Level 2.1, Level 2.2, Level 3.1, Q6 |
20| 29.4 Timeouts | Level 1A.4, Level 3.2, Level 5.1, Q4 |
21| 29.5 Retries | Level 1A.3, Level 1B.2, Level 2.4, Q1 |
22| 29.6 Retry storms | Level 3.3, Level 5.4, Q3 |
23| 29.7 Circuit breakers | Level 1A.1, Level 1B.3, Level 3.4, Level 4, Q2 |
24| 29.8 Bulkheads | Level 1A.2, Level 2.5, Level 3.5, Q5 |
25| 29.9 Graceful degradation | Level 1A.5, Level 5.3, Q9 |
26| 29.10 Health checks and failover | Level 1B.4, Level 5.5, Q7 |
27| 29.11 Disaster recovery | Level 1B.5, Level 2.6, Level 5.5, Q8, Q10 |
28
29---

29. Failures, Retries, Circuit Breakers