29.6 What is a retry storm?

At home: After a power cut, every house in the colony switches everything on at the same moment: ACs, geysers, pumps. The surge trips the transformer again, and the colony stays dark.

In the system: When a service has trouble, its callers' retries add load exactly when it has the least capacity. That can turn a short blip into a long outage that doesn't end on its own, a metastable failure.

Figure: retries turn a 3-second blip into a permanent overload, unless they're budgeted.

In the chapter's code: the service can handle 1,000 requests a second; normal load is 800. A 3-second outage makes every request fail; clients retry (some at two layers), and offered load jumps to 3,200 a second, 4× normal. The outage ends, but the service is now overloaded by retries, so they keep failing and being retried. 47 seconds later it's still stuck at 3,200 offered, serving 1,000 a second, most of them retries of requests users have long given up on. With a retry budget (retries capped at 10% of normal traffic), the same blip is over in one second.

Defences:

• Retry budgets: allow retries only up to a small fraction of normal traffic per client or service.
• Backoff with jitter and one retrying layer.
• Circuit breakers (29.7) so callers stop calling a failing service.
• Load shedding on the server: when overloaded, reject quickly (503 or 429) and cheaply, ideally the least important work first, rather than queueing everything.
• Bounded queues and deadlines, so the server doesn't waste effort on requests whose callers already timed out.

Memory code: "When everyone flicks the switch at once, the lights never come back."

---