23. Pub/Sub and Fan-Out
1
The picture to keep in your head2
23.1 What's the difference between a queue and pub/sub?3
23.2 What are topics and subscribers?4
23.3 What are consumer groups?5
23.4 What are the fan-out strategies?6
23.5 What is event-driven architecture?7
23.6 How do you handle duplicates and ordering?8
23.7 How do you publish events reliably (the outbox pattern)?9
Memory card and where the analogy breaks10
Practice: five levels11
Interview questions12
Solutions13
Sources14
Code lab23. Pub/Sub and Fan-Out
Practice: five levels
3 min read+10 XP
Assignments
Five levels. Do them in order. Answers are in "Solutions" at the end, so try first.
Level 1: Think (no code)
A. Queue or topic? One line each.
- "Resize this uploaded book cover."
- "Order 881 was placed."
- "Book 42's price changed."
- "Send this password-reset email."
- "Author 7 released a new book."
B. True or false? Explain in one line.
- In pub/sub, two instances of the same service in one consumer group both process every message.
- Fan-out on write makes reading a feed cheap.
- Fan-out on read is expensive for celebrities who post.
- "Save the order, then publish the event" can never lose an event.
- With the outbox pattern, subscribers may still receive duplicates.
Level 2: Calculate
- A celebrity author has 5 million followers. With fan-out on write at 50,000 feed writes a second, how long until the last follower's feed has the new book?
- 10,000 new books are released a day, by authors with 200 followers on average. How many feed writes a day with fan-out on write?
- 1 million feed loads a day, and each user follows 300 authors on average. How many author lookups a day with fan-out on read?
- 1,000 orders a second publish
order_placed, and 4 services subscribe. How many deliveries a second does the broker make? - The outbox relay polls every 1 second and publishing takes about 20 ms. What's the worst-case delay between commit and publish?
Level 3: Build
In Python:
- A
Brokerwithsubscribe(topic, group, member)andpublish(topic, msg): every group gets every message; members inside a group share them. FanoutOnWrite,FanoutOnReadandHybridfeed classes withpostandread. Show they return identical feeds and compare their write and read counts.- A
StockViewconsumer that ignores duplicates by event ID and stale events by version. - An outbox: write the order and the outbox row together (all or nothing), and a relay that may publish twice after a crash.
- Reproduce both dual-write failures (ghost event, lost event).
Level 4: Debug
Customers report SMS messages saying "Your order 883 is confirmed!" for orders that don't appear in their account. The checkout code:
python
- Explain exactly how a customer gets an SMS for an order that doesn't exist.
- Moving the publish after the transaction "fixes" that. What new problem does it create?
- Fix it properly.
Level 5: Design
Design the bookstore's "New from authors you follow" feature: 20 million readers, 200,000 authors, most with under 1,000 followers but a few with 5 million; readers open the feed about 5 times a day; about 10,000 books are released a day.
Deliver:
- How a release reaches followers (push, pull, or hybrid, and the threshold).
- Where feeds are stored and how big they get.
- How the release event is published reliably from the catalogue service.
- How duplicates and late events are handled.
- What other services subscribe to
book_released, and how they're kept independent.