Art of Codeartofcode
Lesson 11 of 18

22. Message Queues

1
The picture to keep in your head
2
22.1 What are producers and consumers?
3
22.2 Why go async?
4
22.3 What are the delivery guarantees?
5
22.4 How do acks and visibility timeouts work?
6
22.5 What are retries and dead-letter queues?
7
22.6 What is an idempotent consumer?
8
22.7 How does ordering work?
9
22.8 What is backpressure?
10
22.9 How do you scale consumers?
11
22.10 Log (Kafka) vs queue (RabbitMQ, SQS): what's the difference?
12
22.11 How do delayed and scheduled jobs work?
13
Memory card and where the analogy breaks
14
Practice: five levels
15
Interview questions
16
Solutions
17
Sources
18
Code lab
22. Message Queues

22.10 Log (Kafka) vs queue (RabbitMQ, SQS): what's the difference?

3 min read+10 XP

At the bank: The token board throws away each slip once it's served: one clerk, one job, gone. Now picture instead a ledger: every transaction is written down in order and kept. The cashier reads it, the auditor reads it, the fraud team reads it, each at their own pace, each with their own bookmark. And the auditor can go back to page one.

In the system:

a queue deletes messages after one consumer acks; a log keeps them and each group tracks its own offset.

Figure: a queue hands out jobs; a log keeps a history that many readers can replay.

Queue (RabbitMQ, SQS)Log (Kafka, Kinesis, Pulsar)
After processingMessage deletedMessage kept for a retention period (Kafka defaults to 7 days)
Who gets a messageOne consumerEvery consumer group gets every message
Position trackingBroker tracks per message (acks)Consumer tracks an offset per partition
ReplayNoYes: move the offset back
OrderingUsually none (FIFO options exist)Per partition
Per-message featuresAcks, delays, priorities, DLQs built inFewer; you build retries and DLQs as topics
Sweet spotJobs and tasks: send this email, resize this imageEvent streams: many services reacting to "order placed", analytics, CDC

In the chapter's code, an SMS group and an analytics group both read all five "order placed" events from the same log; the analytics group then rewinds to offset 0 and reads them again.

How teams combine them: Kafka as the backbone of business events ("order placed", "payment captured"), each service consuming what it needs; queues for task work inside a service ("send these 10,000 emails").

Memory code: "A queue hands out slips and bins them; a log keeps the ledger for every reader."