Lesson 7 of 14
23. Pub/Sub and Fan-Out

23.6 How do you handle duplicates and ordering?

3 min read+10 XP

On the paper round: Sometimes a new delivery boy drops a second copy of today's paper. Sometimes yesterday's paper turns up after today's because it was left at the wrong house first. A sensible reader ignores the duplicate and doesn't believe yesterday's headline over today's.

In the system: Pub/sub inherits the delivery realities of Chapter 22:

  • Duplicates: at-least-once delivery means every subscriber may get the same event twice.
  • Ordering: usually guaranteed only per partition or key, not across the topic, and not across different topics.

How subscribers stay correct:

  • Event IDs for de-duplication. Every event has a unique ID; the subscriber records the IDs it has applied (with a unique constraint) and skips repeats.
  • Versions for disorder. Events about the same entity carry an increasing version (or sequence number); the subscriber ignores anything older than what it already has.
  • Key by entity when order matters, so all events for one order or one book land on one partition.

In the chapter's code, a stock view receives "Dune: 10 left (v1)", "Dune: 8 left (v3)", a late "Dune: 9 left (v2)" and a repeat of v3. It applies v1 and v3, marks v2 as stale and the repeat as a duplicate, and ends with the correct 8.

Prefer events that are easy to apply idempotently: "stock for Dune is now 8 (version 3)" is safer than "stock for Dune went down by 2", because applying it twice does no harm.

Memory code: "Skip the second copy, and don't let yesterday's paper beat today's."