30.8 How do you debug an incident, step by step?

When it breaks

In the car: A strange noise on the highway. First, pull over safely. Then: which gauge looks wrong? When did it start? What changed (new tyres yesterday)? Replay the route. Then fix it, and next time, check the tyres after fitting them.

In the system: A practical sequence for an incident like "checkout is slow during the sale":

Figure: stop the bleeding first, then narrow down with data, then learn from it.

1. Mitigate first. Restore service before finding the root cause: roll back the latest deploy, fail over, turn off a feature flag, shed load. Users care that it's fixed, not why.
2. Scope it. What's affected (which SLIs, which journeys)? Since when? How many users? Declare an incident, assign a lead and a communicator, start a timeline.
3. Break it down by dimension. Split error rate and latency by version, region, instance, endpoint, customer type, payment method, dependency. The problem usually concentrates somewhere. In the chapter's code, splitting checkout 5xx errors by version shows v41 at 0% and v42 at about 21%, while all three regions look alike: the new release is the cause.
4. Follow one bad request. Pick an example trace from the bad slice, find the slow or failing span (30.2.3), read its logs by trace ID.
5. Ask "what changed?" Deploys, config, feature flags, traffic patterns, dependencies, certificates, quotas. Deploy markers on graphs help.
6. Fix and verify. Confirm the SLIs return to normal, not just that the error message stopped.
7. Blameless postmortem. Write down the timeline, impact, root cause and contributing factors, what went well, and action items (better alert, safer rollout, missing test). Focus on systems and processes, not individuals.

Over time, the postmortems tell you where to invest: if most incidents start with deploys, invest in canary releases and automatic rollback.

Memory code: "Pull over, read the gauges, find what changed, replay the trip, then fix and write it down."

---