1Show the Level 1 solution
2**A.**
31. **Metric.**
42. **Logs** (found via her trace or order ID), plus the **trace**.
53. **Trace.**
64. **Metric** (a counter's rate).
75. **Log.**
89**B.**
101. **False.** It hides the tail; use percentiles or an SLO-based latency SLI.
112. **False.** Merge histograms, then compute.
123. **False.** It explodes cardinality; use logs and traces.
134. **False.** The SLA should be looser, so you fix problems before owing penalties.
145. **False.** It's a cause, not a user symptom; make it a ticket or dashboard item.
1Show the Level 2 solution
21. 0.1% × 20,000,000 = **20,000** failed checkouts. As full outage time: 0.1% of 30 days = **43.2 minutes**.
32. 1.44% ÷ 0.1% = burn rate **14.4**; 30 days ÷ 14.4 ≈ **50 hours** until the budget is gone.
43. 20 × 5 × 3 = **300** series; with 10 versions, **3,000**.
54. Average of p99s = (100 + 5,000) ÷ 2 = **2,550 ms**. The real p99 over all 10,000 requests is about **5,000 ms**: the slowest 1% (100 requests) all come from server B's slow requests, and averaging percentiles ignores how many requests each server served and how their distributions combine.
65. 5 KB × 2,000 × 86,400 ≈ **864 GB** a day. Sampled: about 10% of info logs + all errors ≈ 864 × (0.1 × 0.99 + 0.01) ≈ **94 GB** a day.
1Show the Level 3 solution
2Tested solution (Python 3), the key parts:
34```python
5class Histogram:
6 def __init__(self):
7 self.counts, self.total = [0] * (len(BUCKETS) + 1), 0
89 def observe(self, ms):
10 self.counts[bisect.bisect_left(BUCKETS, ms)] += 1
11 self.total += 1
1213 def merge(self, other):
14 out = Histogram()
15 out.counts = [a + b for a, b in zip(self.counts, other.counts)]
16 out.total = self.total + other.total
17 return out
1819 def quantile(self, q):
20 target, seen = q * self.total, 0
21 for i, c in enumerate(self.counts):
22 seen += c
23 if seen >= target:
24 return BUCKETS[i] if i < len(BUCKETS) else float("inf")
252627def self_time(spans):
28 children = defaultdict(list)
29 for name, parent, s, e in spans:
30 if parent:
31 children[parent].append((s, e))
32 return {name: (e - s) - sum(ce - cs for cs, ce in children[name])
33 for name, parent, s, e in spans}
343536def burn_rate(error_ratio, slo):
37 return error_ratio / (1 - slo)
383940def should_page(err_1h, err_5m, slo, threshold=14.4):
41 return burn_rate(err_1h, slo) >= threshold and burn_rate(err_5m, slo) >= threshold
42```
4344All checks pass: logs filter by trace ID and level; the mean (226 ms) and median (140 ms) hide a 4.4 s p99; averaged p99s give 2,550 ms while merged histograms give 5,000 ms; adding `user_id` turns 300 series into 300 million; the trace's slowest self time is 1,540 ms in `gateway.http`; the error budget is 20,000 checkouts and 1.44% errors burns at 14.4×; the multiwindow rule pages on a live incident but not after recovery or on a slow burn; and errors break down to v42 (about 21%) rather than any region. The full file is in `code/ch30_observability.py`.
1Show the Level 4 solution
21. **Why it stayed quiet:** it **averages p99s** across 20 servers. 18 healthy servers at about 300 ms and 2 sick ones at 5,000 ms average to (18 × 300 + 2 × 5,000) ÷ 20 = **770 ms**, well under 2,000. Yet **10%** of all checkouts (2 of 20 servers' share) were taking 5 seconds, far beyond any reasonable SLO. Averaging percentiles hides exactly this kind of partial failure.
32. **The fix:** alert on the **user-facing SLI** computed from **all requests** (or merged histograms), with burn-rate rules:
45```python
6# SLI: fraction of checkouts that succeed within 2 s, from load-balancer data (all servers merged)
7good = count(checkouts, status="2xx", duration_ms__lte=2000, window="1h")
8total = count(checkouts, window="1h")
9err_1h = 1 - good / total
10err_5m = 1 - count(..., window="5m") / count(checkouts, window="5m")
11if should_page(err_1h, err_5m, slo=0.999):
12 page("checkout SLO burning fast")
13```
1415With 10% of checkouts over 2 s, the error ratio is about 10%, a burn rate of about **100**: it pages within minutes. Separately, a **ticket-level** alert (or automatic ejection by the load balancer's outlier detection) for any single instance whose p99 is far above its peers.
1Show the Level 5 (sketch) solution
21. **SLIs/SLOs**, measured at the API gateway (and from real-user monitoring on clients):
3 - Checkout availability: successful checkout API calls ÷ valid calls, **99.9%** over 30 days.
4 - Checkout latency: calls completing within **2 s** ÷ all calls, **99%**.
5 - Order confirmation SMS: sent within **2 minutes** of order, **99%**.
62. **Telemetry:**
7 - **Logs:** JSON with timestamp, level, service, version, region, trace ID, order ID, payment method, gateway, error code; no card data or full addresses; info sampled at 10%, errors kept.
8 - **Metrics:** RED per service and per dependency (labels: endpoint, status class, region, version, payment method, gateway: all bounded); USE for DB pool, Kafka consumer lag and oldest-message age.
9 - **Traces:** OpenTelemetry everywhere, trace context through the gateway, gRPC and Kafka headers; 5% head sampling plus all errors and slow (> 2 s) traces.
103. **Alerts:**
11 - **Page:** checkout availability or latency SLO burning at ≥ 14.4× over 1 h and 5 min; SMS SLO burning fast.
12 - **Ticket:** burn ≥ 1× over 3 days; DLQ not empty; a single instance far slower than its peers; DB replication lag rising.
13 - No pages on CPU or memory alone.
144. **Dashboard (top to bottom):** SLI vs SLO and error budget remaining; checkouts per minute, error rate, p50/p95/p99 with deploy markers; breakdowns by platform, version, region and payment method; dependency panels (pricing, inventory, payment gateway latency and errors per gateway and method); resources (DB pool, Kafka lag, instance health).
155. **Finding "UPI only":** the latency SLO page fires; the dashboard's breakdown by **payment method** shows UPI p99 at 6 s while cards are normal; the dependency panel shows the gateway's UPI endpoint slow; an example trace from that slice shows the time in `gateway.http` for UPI collect requests; logs for that trace show gateway timeouts. Mitigate by routing UPI to the backup gateway or showing other methods, and open a ticket with the provider.