Art of Codeartofcode
Lesson 4 of 14

23. Pub/Sub and Fan-Out

1
The picture to keep in your head
2
23.1 What's the difference between a queue and pub/sub?
3
23.2 What are topics and subscribers?
4
23.3 What are consumer groups?
5
23.4 What are the fan-out strategies?
6
23.5 What is event-driven architecture?
7
23.6 How do you handle duplicates and ordering?
8
23.7 How do you publish events reliably (the outbox pattern)?
9
Memory card and where the analogy breaks
10
Practice: five levels
11
Interview questions
12
Solutions
13
Sources
14
Code lab
23. Pub/Sub and Fan-Out

23.3 What are consumer groups?

3 min read+10 XP

On the paper round: The Hindu has a thousand subscribers in one area. One delivery boy can't do them all, so four boys share the area. Each house still gets one copy, not four. Meanwhile, a separate team delivers Sportstar to its own subscribers.

In the system: A consumer group (Kafka), or a subscription with several workers (Google Pub/Sub, SQS behind SNS), lets several instances of one service share the messages of a subscription:

  • Between groups: every group gets every message (pub/sub).
  • Inside a group: each message goes to one member (a queue).

one topic, three groups; inside the SMS group, two workers split the messages.

Figure: pub/sub between groups, a queue inside each group.

In the chapter's code, 10 "order placed" events reach the loyalty and analytics groups in full, while the SMS group's two workers get 5 each, together covering all 10.

What this gives you:

  • Scale a service by adding members to its group.
  • Add a new service by adding a new group; nobody else changes.
  • In Kafka, a group's members split the partitions, so a topic's partition count caps how many members can work at once (Chapter 22).

Memory code: "Every team gets the paper; inside a team, the boys split the street."