29. Failures, Retries, Circuit Breakers
The picture to keep in your head
Every question below is answered twice: first at home, so you can picture it, then in the system, so you know exactly how it works. Each answer ends with a memory code to lock it in.
The picture to keep in your head
Your house has a fuse box. When the AC develops a fault and starts drawing too much current, its trip switch cuts that one circuit. The fridge, the lights and the Wi-Fi stay on, because they're on separate circuits. You don't flick the switch back on fifty times in a row; you wait a bit, try once, and if it trips again, you leave it off and call the electrician. You keep spare bulbs so no single bulb leaves a room dark. And when the power cut comes, the inverter keeps the lights and fans running, though not the AC.
That's this chapter: things will fail, so contain the damage. Trip the bad circuit, keep circuits separate, retry calmly, keep spares, and when power is short, run the essentials.
Our running example: the bookstore's checkout. It calls the payment gateway and the inventory service, and the page's "you may also like" box calls recommendations. Availability, latency and traffic numbers are assumptions for practice.
| At home | In the system |
|---|---|
| Bulbs blow, wires fray, the grid cuts out | Kinds of failure |
| Spare bulbs; two water pumps | Redundancy, no single point of failure |
| "Power was out 9 hours this year" | Availability nines |
| "If the kettle hasn't boiled in 5 minutes, check it" | Timeouts |
| Wait, then try the switch again | Retries with backoff and jitter |
| Everyone flicking switches at once after a cut | Retry storms |
| The trip switch | Circuit breakers |
| Separate circuits for AC, fridge and lights | Bulkheads |
| The inverter: lights on, AC off | Graceful degradation |
| Checking the meter; switching to the inverter | Health checks and failover |
| Insurance and a second home | Disaster recovery |
Figure: contain the damage so one bad circuit doesn't black out the house.