Art of Codeartofcode
Lesson 8 of 14

23. Pub/Sub and Fan-Out

1
The picture to keep in your head
2
23.1 What's the difference between a queue and pub/sub?
3
23.2 What are topics and subscribers?
4
23.3 What are consumer groups?
5
23.4 What are the fan-out strategies?
6
23.5 What is event-driven architecture?
7
23.6 How do you handle duplicates and ordering?
8
23.7 How do you publish events reliably (the outbox pattern)?
9
Memory card and where the analogy breaks
10
Practice: five levels
11
Interview questions
12
Solutions
13
Sources
14
Code lab
23. Pub/Sub and Fan-Out

23.7 How do you publish events reliably (the outbox pattern)?

3 min read+10 XP

On the paper round: When you subscribe, the agent writes it in the order book at the counter, and the delivery list is made from the book each night. If the agent instead promised to "remember to tell the press later", a busy day would mean some subscriptions are simply forgotten.

In the system: A service usually needs to change its database and publish an event together: save order 881 and publish order_placed. These are two different systems, so doing both directly (a dual write) can go wrong either way:

  • Publish, then commit: if the commit then fails, subscribers react to an order that doesn't exist (a "ghost" event). Customers get an SMS for an order that was never placed.
  • Commit, then publish: if the service crashes in between, the order exists but the event is lost. No SMS, no points, ever.

In the chapter's code, both failures are reproduced: one leaves a ghost event, the other a silent gap.

The transactional outbox fixes it:

  1. In one database transaction, write the order and a row in an outbox table describing the event.
  2. A separate relay reads unsent outbox rows, publishes them to the broker, and marks them sent. Or a change data capture tool (such as Debezium) reads the outbox rows straight from the database log.

orders and outbox written in one transaction; a relay or CDC publishes outbox rows to the broker.

Figure: the event exists if and only if the order does.

Properties:

  • Atomic: if the transaction rolls back, there's no outbox row and no event. If it commits, the event will be published eventually.
  • At-least-once: if the relay crashes after publishing but before marking the row, it publishes again on restart. In the chapter's code, the event for order 881 is published twice; subscribers must de-duplicate by event ID (23.6).
  • Ordering: publish outbox rows in insertion order per entity.
  • Housekeeping: delete or archive sent rows so the table doesn't grow forever.

The mirror-image pattern on the consumer side is the inbox: record processed event IDs in the same transaction as the effect (Chapter 22's idempotent consumer).

Memory code: "Write it in the order book with the order; the relay posts it later."