23. Pub/Sub and Fan-Out
23.6 How do you handle duplicates and ordering?
On the paper round: Sometimes a new delivery boy drops a second copy of today's paper. Sometimes yesterday's paper turns up after today's because it was left at the wrong house first. A sensible reader ignores the duplicate and doesn't believe yesterday's headline over today's.
In the system: Pub/sub inherits the delivery realities of Chapter 22:
- Duplicates: at-least-once delivery means every subscriber may get the same event twice.
- Ordering: usually guaranteed only per partition or key, not across the topic, and not across different topics.
How subscribers stay correct:
- Event IDs for de-duplication. Every event has a unique ID; the subscriber records the IDs it has applied (with a unique constraint) and skips repeats.
- Versions for disorder. Events about the same entity carry an increasing version (or sequence number); the subscriber ignores anything older than what it already has.
- Key by entity when order matters, so all events for one order or one book land on one partition.
In the chapter's code, a stock view receives "Dune: 10 left (v1)", "Dune: 8 left (v3)", a late "Dune: 9 left (v2)" and a repeat of v3. It applies v1 and v3, marks v2 as stale and the repeat as a duplicate, and ends with the correct 8.
Prefer events that are easy to apply idempotently: "stock for Dune is now 8 (version 3)" is safer than "stock for Dune went down by 2", because applying it twice does no harm.
Memory code: "Skip the second copy, and don't let yesterday's paper beat today's."