30. Observability
30.3 What are golden signals, RED and USE?
In the car: Dozens of sensors, but a good dashboard shows a handful: speed, fuel, temperature, and warnings. Those few tell you almost everything you need while driving.
In the system: Three well-known short lists of what to watch:
The four golden signals (Google SRE book), for any user-facing service:
- Latency: how long requests take (separate successful and failed requests; a fast error isn't a fast success).
- Traffic: how much demand (requests per second, checkouts per minute).
- Errors: how many requests fail (explicitly with 5xx, or implicitly with wrong content or too slow).
- Saturation: how full the service is (CPU, memory, thread pools, queue depth): the warning before latency rises.
RED (Tom Wilkie), a compact version for request-driven services:
- Rate, Errors, Duration.
USE (Brendan Gregg), for resources (CPU, disks, network, connection pools):
- Utilisation: how busy it is.
- Saturation: how much work is waiting.
- Errors: error events.
For the bookstore: every service gets a RED dashboard (checkout: checkouts per second, failed checkouts, p50/p99 checkout time); the database, cache and queues get USE views (connection pool utilisation, query queue, replication lag, consumer lag, Chapter 22). Start here before inventing custom metrics.
Memory code: "Speed, fuel, temperature, warnings: watch the few that tell you most."