24.9 How do you retry safely?

At the machine: If the screen freezes, wait a moment and tap again with the same receipt. Don't hammer the screen ten times a second, and don't keep retrying if it says "card declined": that won't change.

In the system: Retries turn temporary failures into successes, but bad retries cause duplicate effects and retry storms that make outages worse. Rules:

1. Retry only what can succeed next time.

• Retry: timeouts, connection errors, 429, 500, 502, 503, 504.
• Don't retry: 400, 401, 403, 404, 409, 422. The same request will fail the same way.

2. Retry only what's safe to repeat.

• GET, PUT, DELETE: safe by design.
• POST: only with an idempotency key. In the chapter's code, should_retry("POST", 503, has_key=False) is False; with a key, it's True.

3. Back off, with jitter. Wait longer each time (0.5 s, 1 s, 2 s, 4 s ...) up to a cap, with a random spread so thousands of clients don't retry in lockstep (Chapter 21's thundering herd). Respect a Retry-After header when the server sends one.

4. Limit retries. A few attempts, and an overall deadline ("give up after 10 seconds"). Use a retry budget across a service (for example, retries may add at most 10% extra traffic), and circuit breakers to stop calling something that's clearly down (Chapter 29).

5. Retry at one layer. If the browser, the API gateway, the service and its HTTP library each make 3 attempts, one failure can become 3 × 3 × 3 × 3 = 81 calls.

Memory code: "Retry the temporary, only with a receipt, slower each time, and not forever."

---