30.2 What are the three pillars?

In the car: Three ways to understand a journey: the log of events, the gauges over time, and the route replay of one trip.

In the system: The classic three kinds of telemetry:

Figure: logs tell you what happened, metrics tell you how much and how often, traces tell you where.

They work best linked together: a spike on a metric graph → example traces from that minute → the logs for that trace ID. Standards like OpenTelemetry let one library emit all three with shared context.

30.2.1 Logs

In the car: The trip log: "10:42:13, hard brake, speed 82 km/h". Each entry is one event, with details.

In the system: A log is a timestamped record of an event. Make logs structured (JSON with named fields), not free text, so they can be searched and counted:

In the chapter's code, structured logs can be filtered by trace_id to see one checkout's full story, or by level to find every error.

Good practice:

• Include context: request or trace ID, user or order ID where allowed, service, version, region.
• Use levels (debug, info, warn, error) and keep debug off in production by default.
• Never log secrets or sensitive personal data (passwords, card numbers, full addresses); mask them.
• Control volume: logs are the most expensive pillar at scale. Sample noisy info logs, keep all errors, and set retention by value.
• Ship them centrally (an ELK/OpenSearch stack, Loki, or a hosted service) so you're not SSH-ing into servers.

30.2.2 Metrics

In the car: The gauges: speed, fuel, temperature, shown continuously. You don't need a log entry for every revolution of the engine to know it's overheating.

In the system: A metric is a number measured over time, aggregated: requests per second, errors per second, latency distribution, queue depth, CPU. Common types:

• Counter: only goes up (requests served, errors); you graph its rate.
• Gauge: goes up and down (queue depth, memory in use, connections).
• Histogram: counts observations in buckets (requests that took 0-25 ms, 25-50 ms ...), so you can compute percentiles.

Two traps the chapter's code demonstrates:

• Averages hide the tail. With 98% of checkouts at 80-200 ms and 2% at 3-6 seconds, the mean is about 226 ms and the median about 140 ms: both look fine. The p99 is about 4.4 seconds: one customer in a hundred waits that long. Watch p50, p95, p99.
• You can't average percentiles. Server A (9,000 requests, p99 100 ms) and server B (1,000 requests, p99 5,000 ms): the average of their p99s is 2,550 ms, but the true p99 across both is 5,000 ms. Keep histograms per server and add the buckets, then compute percentiles.

Figure: look at percentiles, and merge histograms instead of averaging percentiles.

Watch cardinality: each unique combination of label values is a separate time series. A latency metric with 20 endpoints × 5 status classes × 3 regions is 300 series; add user_id with a million users and it's 300 million, which will break your metrics system. Put high-cardinality details (user, order, trace ID) in logs and traces, not metric labels. Prometheus is the common open-source choice, with Grafana for graphs.

30.2.3 Traces

In the car: The route replay: one specific trip, showing exactly how long each stretch took, and that the twenty minutes were lost at the toll plaza.

In the system: A distributed trace follows one request through every service it touches. It's a tree of spans: each span has a name, a start and end time, a parent, and attributes. A trace ID is created at the edge and passed along in a header (the W3C traceparent header) to every downstream call, so all spans join up.

Figure: the trace shows where the time went.

In the chapter's code, one 1,840 ms checkout trace breaks down by self time (time in a span not spent in its children): 1,540 ms is gateway.http, waiting on the payment gateway. Checkout's own code, auth, cart, pricing and inventory together take under 250 ms. No amount of staring at checkout's CPU graph would have shown that.

Practicalities:

• Sample: keep a fraction of traces (say 1 to 10%), plus all slow or failed ones (tail-based sampling).
• Instrument at the edges: HTTP and gRPC clients and servers, database drivers, queue producers and consumers (pass trace context in message headers, Chapters 22 and 23).
• Put the trace ID in every log line so you can jump from a trace to its logs.
• Tools: OpenTelemetry for instrumentation; Jaeger, Tempo, Zipkin or hosted services for storage and viewing.

Memory code: "The log says what, the gauges say how much, the replay says where."