Art of Codeartofcode
Lesson 6 of 15

30. Observability

1
The picture to keep in your head
2
30.1 Why do you need observability?
3
30.2 What are the three pillars?
4
30.3 What are golden signals, RED and USE?
5
30.4 What are SLI, SLO and SLA?
6
30.5 What are error budgets?
7
30.6 How do you alert without noise?
8
30.7 What makes a good dashboard?
9
30.8 How do you debug an incident, step by step?
10
Memory card and where the analogy breaks
11
Practice: five levels
12
Interview questions
13
Solutions
14
Sources
15
Code lab
30. Observability

30.5 What are error budgets?

3 min read+10 XP

In the car: If the service is due every 10,000 km and you're at 7,500, you have 2,500 km of slack. You can take a long road trip. If you're at 9,800, you skip the trip and book the service.

In the system: An error budget is the amount of unreliability your SLO allows: 1 − SLO. With a 99.9% checkout SLO and 20 million checkouts a month, the budget is 0.1%, about 20,000 failed checkouts a month (computed in the chapter's code). If the month so far has 19,985,000 good checkouts out of 20 million, the SLI is 99.925%: within the SLO, with budget left.

How teams use it:

  • Budget remaining: ship features, run experiments, do risky migrations.
  • Budget nearly spent: slow down releases, prioritise reliability work, add tests and safeguards.
  • It ends arguments: product and engineering agree in advance on how much unreliability is acceptable, so "should we ship?" becomes a number, not a debate.

Burn rate is how fast you're spending the budget compared with the steady rate that would use exactly all of it by the end of the window:

  • burn rate = current error ratio ÷ (1 − SLO).
  • With a 99.9% SLO, a 1.44% error ratio is a burn rate of 14.4: at that pace, a 30-day budget is gone in about 50 hours.

Burn rate is the bridge from SLOs to alerts (30.6).

Memory code: "The slack before the next service is yours to spend, but watch how fast you're spending it."