Practice: five levels

Assignments

Five levels. Do them in order. Answers are in "Solutions" at the end, so try first.

Level 1: Think (no code)

A. Log, metric or trace? One line each.

1. "Checkouts per second over the last week."
2. "Why did Asha's checkout at 10:42 fail?"
3. "Which service made this request take 1.8 seconds?"
4. "How many 5xx errors per minute from the payment client?"
5. "The exact error message and gateway response for order 881."

B. True or false? Explain in one line.

1. Average latency is the best single latency number to alert on.
2. You can average the p99 latencies of two servers to get the overall p99.
3. Adding user_id as a metric label is a good way to debug per-user issues.
4. An SLA should be stricter than the internal SLO.
5. "CPU above 80% on one server" is a good reason to page someone at night.

Level 2: Calculate

1. Checkout SLO: 99.9% over 30 days, with 20 million checkouts. What's the error budget in failed checkouts? In "minutes of full outage"?
2. Right now 1.44% of checkouts fail. What's the burn rate, and how long until the month's budget is gone?
3. A metric has labels endpoint (20 values), status class (5) and region (3). How many time series? What if you add app version (10 values)?
4. Server A: 9,000 requests, p99 100 ms. Server B: 1,000 requests, p99 5,000 ms. What's the average of their p99s, and why is the real p99 different?
5. A service logs 5 KB per request at 2,000 requests a second. How much log data per day? With info logs sampled at 10% (errors kept, assume 1%)?

Level 3: Build

In Python:

1. log_line(level, msg, **fields) producing JSON, and query_logs(lines, **match).
2. A Histogram with fixed buckets, observe, merge and quantile; show averaging p99s understates the merged p99.
3. self_time(spans) for a trace, and find the span where most time went.
4. error_budget, burn_rate and a multiwindow should_page(err_1h, err_5m, slo).
5. breakdown(requests, dim) that shows error rates by version and by region.

Level 4: Debug

During the sale, customers complained about slow checkouts for 40 minutes before anyone noticed. The alert:

Two of the twenty servers (on a bad host) were taking 5 seconds for most requests, and they were receiving a normal share of traffic.

1. Explain why this alert didn't fire.
2. Rewrite the alerting approach so it would have fired, and say what you'd page on.

Level 5: Design

Design observability for the bookstore's checkout journey: web and mobile clients, an API gateway, checkout, cart, pricing, inventory, payments (calling an external gateway), an order database, and Kafka events to SMS and loyalty services. Peak: 1,000 checkouts a minute.

Deliver:

1. The SLIs and SLOs for checkout, and where you measure them.
2. What each service emits (logs, metrics, traces), including what's in a log line and which metric labels are allowed.
3. Your alerting rules: what pages, what becomes a ticket.
4. The checkout dashboard, top to bottom.
5. How an on-call engineer would find that "the payment gateway is slow for UPI payments only" within 10 minutes.