Art of Codeartofcode
Lesson 6 of 18

22. Message Queues

1
The picture to keep in your head
2
22.1 What are producers and consumers?
3
22.2 Why go async?
4
22.3 What are the delivery guarantees?
5
22.4 How do acks and visibility timeouts work?
6
22.5 What are retries and dead-letter queues?
7
22.6 What is an idempotent consumer?
8
22.7 How does ordering work?
9
22.8 What is backpressure?
10
22.9 How do you scale consumers?
11
22.10 Log (Kafka) vs queue (RabbitMQ, SQS): what's the difference?
12
22.11 How do delayed and scheduled jobs work?
13
Memory card and where the analogy breaks
14
Practice: five levels
15
Interview questions
16
Solutions
17
Sources
18
Code lab
22. Message Queues

22.5 What are retries and dead-letter queues?

3 min read+10 XP

At the bank: Some customers can't be served at the counter: a missing document, a strange request. The clerk tries, then tries again. After three tries, the token goes to the manager's desk, so the counters keep moving. Someone looks at those cases by hand.

In the system: Failures come in two kinds:

  • Transient: the SMS gateway timed out, the database was briefly overloaded. Retrying later usually works.
  • Permanent: the message is malformed, the phone number doesn't exist, the code has a bug for this input. Retrying never works. These are poison messages.

Retries should back off, so a struggling downstream gets breathing room: wait 2 s, 4 s, 8 s, 16 s, 32 s, up to a cap, with jitter (Chapter 21, Chapter 29). Brokers do this with delayed redelivery, or you re-enqueue with a delay.

A dead-letter queue (DLQ) catches messages that fail too many times. In SQS, a redrive policy moves a message to the DLQ after maxReceiveCount receives; RabbitMQ has dead-letter exchanges. In the chapter's code, a message with a malformed address is tried 3 times and then moved to the DLQ, and the main queue keeps flowing.

What to do with a DLQ:

  • Alert when it's not empty. A DLQ nobody watches is a slow data-loss bug.
  • Inspect the messages, fix the bug or the data.
  • Redrive: move them back to the main queue once fixed.
  • Keep enough retention to fix things (SQS keeps messages 4 days by default, up to 14).

Memory code: "Try again with growing pauses; after a few, send it to the manager's desk."