22. Message Queues
1
The picture to keep in your head2
22.1 What are producers and consumers?3
22.2 Why go async?4
22.3 What are the delivery guarantees?5
22.4 How do acks and visibility timeouts work?6
22.5 What are retries and dead-letter queues?7
22.6 What is an idempotent consumer?8
22.7 How does ordering work?9
22.8 What is backpressure?10
22.9 How do you scale consumers?11
22.10 Log (Kafka) vs queue (RabbitMQ, SQS): what's the difference?12
22.11 How do delayed and scheduled jobs work?13
Memory card and where the analogy breaks14
Practice: five levels15
Interview questions16
Solutions17
Sources18
Code lab22. Message Queues
22.6 What is an idempotent consumer?
3 min read+10 XP
At the bank: If your token is called twice by mistake, the second clerk checks the stamp in your passbook: "already paid". They don't pay you twice.
In the system: With at-least-once delivery, every consumer will sometimes see the same message twice. An idempotent consumer produces the same result however many times it processes a message.
Some operations are naturally idempotent:
- "Set order 881's status to packed": doing it twice leaves it packed.
- "Upsert the search document for book 42."
Others aren't, and need protection:
- "Add 50 points to Asha": twice gives 100.
- "Send an SMS": twice annoys the customer.
- "Charge ₹499": twice is a real problem.
How to make them idempotent:
- Record what you've processed. Keep a table of processed message IDs, or better, a business key like
(order_id, "loyalty_points"), with a unique constraint. Insert the record and apply the effect in the same transaction (Chapter 17); a duplicate fails the insert and is skipped. - Use conditional updates:
UPDATE orders SET status='packed' WHERE id=881 AND status='paid'. - Pass idempotency keys downstream: when the worker calls the payment provider or SMS gateway, send the order ID as the idempotency key, so they de-duplicate too (Chapter 24).
In the chapter's code, order 881's points message is delivered twice. The naive worker gives Asha 100 points; the idempotent one gives 50.
Memory code: "Stamp the passbook, so a second call pays nothing."