Interview questions

Interview questions (with the counter-questions you'll get)

Q1. What's the difference between monitoring and observability?

They're testing: concepts beyond buzzwords.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Monitoring tracks known conditions with predefined checks and dashboards. Observability is having rich enough telemetry, with context like trace IDs, versions and regions, to answer new questions about why the system behaves as it does, without shipping new code. In practice: structured logs, metrics and traces, linked together.
3
4**Counter-questions:**
5- *"Give a question only observability answers."* → "Why is checkout slow only for UPI payments on the newest Android version?"
6
7**Trap:** "observability is just more dashboards".

Q2. Explain logs, metrics and traces, and when to use each.

They're testing: the three pillars.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Logs are timestamped event records with detail, best for investigating specific events. Metrics are aggregated numbers over time, cheap to store and query, best for trends, dashboards and alerts. Traces follow one request across services as timed spans, best for finding where latency or errors come from. Link them with trace IDs.
3
4**Counter-questions:**
5- *"Which is most expensive?"* → Usually logs at high volume; sample and set retention.
6
7**Trap:** using logs for everything.

Q3. Why use percentiles instead of averages for latency?

They're testing: statistical literacy.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Latency distributions have long tails; a few very slow requests barely move the mean but ruin those users' experience. p95 and p99 show the tail. Percentiles can't be averaged across servers; compute them from merged histograms.
3
4**Counter-questions:**
5- *"Why does p99 matter if it's only 1%?"* → A page with 20 backend calls hits a p99 path often, and the slowest users are often the most active ones.
6
7**Trap:** alerting on average latency.

Q4. What are SLIs, SLOs and SLAs?

They're testing: reliability vocabulary.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** An SLI is a measurement of user experience, like the fraction of successful checkouts under 2 seconds. An SLO is the target for it over a window, like 99.9% over 30 days. An SLA is an external contract with penalties, set looser than the SLO.
3
4**Counter-questions:**
5- *"Why not 100%?"* → Impossibly costly, and users can't perceive it through their own networks; it also blocks all change.
6
7**Trap:** confusing SLO with SLA.

Q5. What is an error budget, and how is it used?

They're testing: SRE practice.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** One minus the SLO: the allowed unreliability over the window. When there's budget left, ship features and take risks; when it's nearly spent, slow releases and invest in reliability. Burn rate measures how fast it's being consumed and drives alerting.
3
4**Counter-questions:**
5- *"Who decides?"* → Product and engineering agree on the SLO and the policy in advance.
6
7**Trap:** treating the budget as a target to spend fully.

Q6. How do you design alerts that don't cause fatigue?

They're testing: on-call maturity.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Page only on user-facing symptoms that need urgent action, using SLO burn-rate alerts over multiple windows; turn everything else into tickets or dashboards. Make every alert actionable with a runbook, route to owners, deduplicate, and regularly delete alerts that fire without needing action.
3
4**Counter-questions:**
5- *"Why two windows?"* → The long window shows the burn is significant; the short one confirms it's still happening, so the alert resets quickly after a fix.
6
7**Trap:** paging on CPU thresholds.

Q7. How does distributed tracing work?

They're testing: mechanics.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** The first service creates a trace ID and a root span; every outgoing call carries the trace context in headers (W3C traceparent); each service records spans with timing and attributes and sends them to a collector, which assembles the tree. Traces are sampled, keeping errors and slow requests preferentially, and the trace ID is added to logs.
3
4**Counter-questions:**
5- *"What about async messages?"* → Put trace context in message headers and link consumer spans to the producer's.
6
7**Trap:** not knowing how context is propagated.

Q8. What is metric cardinality, and why does it matter?

They're testing: operating metrics systems.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Each unique combination of label values is a separate time series. High-cardinality labels like user ID or order ID multiply series into millions, overwhelming memory and storage and slowing queries. Keep metric labels to bounded sets (endpoint, status class, region, version) and put high-cardinality details in logs and traces.
3
4**Counter-questions:**
5- *"How do you debug per-user issues then?"* → Logs and traces with user IDs, queried on demand.
6
7**Trap:** labelling by user ID.

Q9. Walk me through how you'd debug a production incident.

They're testing: calm, structured thinking.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Mitigate first (roll back, fail over, disable a flag). Scope impact via SLIs. Break errors and latency down by dimensions like version, region and dependency to find where the problem concentrates. Look at example traces and their logs. Check what changed. Fix, verify the SLIs recover, and run a blameless postmortem with action items.
3
4**Counter-questions:**
5- *"When do you roll back without knowing the cause?"* → When the timing lines up with a deploy and rollback is safe: restore service first, investigate after.
6
7**Trap:** debugging the root cause while users are still affected.

Q10. What would you monitor for a message queue consumer?

They're testing: applying RED/USE beyond HTTP.

Think about your answer first, then open it.

reveal

1Show the strong answer, counter-questions and trap
2**Strong answer:** Consumer lag (messages behind, and age of the oldest message), processing rate and error rate, processing duration percentiles, retries and dead-letter queue size, plus consumer instance health. Alert on lag age against an SLO like "SMS sent within 2 minutes", not on raw depth.
3
4**Counter-questions:**
5- *"Why age over depth?"* → Depth depends on traffic; age directly measures how late the work is (Chapter 22).
6
7**Trap:** only watching CPU of consumer machines.
8
9---
10
11## Practice coverage
12
13Every section in this chapter is tested at least once.
14
15| Section | Tested by |
16|---|---|
17| 30.1 Why observability | Q1 |
18| 30.2 Three pillars | Level 1A, Q2 |
19| 30.2.1 Logs | Level 1A.2, Level 1A.5, Level 2.5, Level 3.1, Q2 |
20| 30.2.2 Metrics | Level 1B.1, Level 1B.2, Level 1B.3, Level 2.3, Level 2.4, Level 3.2, Level 4, Q3, Q8 |
21| 30.2.3 Traces | Level 1A.3, Level 3.3, Q7 |
22| 30.3 Golden signals, RED, USE | Level 5.2, Q10 |
23| 30.4 SLI, SLO, SLA | Level 1B.4, Level 5.1, Q4 |
24| 30.5 Error budgets | Level 2.1, Level 2.2, Level 3.4, Q5 |
25| 30.6 Alerting | Level 1B.5, Level 4, Level 5.3, Q6 |
26| 30.7 Dashboards | Level 5.4 |
27| 30.8 Debugging an incident | Level 3.5, Level 5.5, Q9 |
28
29---

30. Observability