33.7 How do retries and de-duplication work?

Delivery is at-least-once inside the system: queues can redeliver a message after a worker crash or a visibility timeout (Chapter 22), and API callers retry after timeouts (Chapter 24). Without care, users get two SMS. Defences, layered:

1. At the API: the caller's idempotency_key is unique in the notifications table; a repeated call returns the same notification ID instead of creating another.
2. In the worker: before calling a provider, check a sent-record for the message's key (per user, type and channel). If it's there, skip. In the chapter's code, the queue redelivers the OTP message; the second delivery is skipped, and only one SMS goes out.
3. At the provider: pass the key to providers that support idempotency.

Retries: on timeouts and 5xx, retry with exponential backoff and jitter (Chapter 29), within the message's deadline. Fail over between providers: in the chapter's code, provider A times out twice and the retry goes to provider B on attempt 2. After the attempt limit, the message goes to a dead-letter queue, the status becomes "failed", and an alert fires if the failure rate rises.

Don't retry what's stale: an OTP picked up after its 5-minute expiry is dropped, not sent. An "order shipped" push sent three days late does more harm than good.

Interviewer pushback:

• "The provider timed out, but did it send?" → We can't know (Chapter 24's lost reply). Retrying the same provider with the same key is safe if it de-duplicates; failing over to another provider risks a duplicate. For OTPs we accept a rare duplicate over a missing code; for marketing we don't fail over at all.

Memory code: "Retry with the same key, skip what's already sent, drop what's gone stale."