33. Design a Notification System
33.6 Why one queue per channel?
Step 5: Deep dives (10 minutes)
Figure: separate lanes so bulk traffic can never delay urgent traffic.
Channels differ in speed, limits and failure modes: push is fast and cheap, SMS is slow-ish and rate-limited by the provider, email is cheap but has its own throttling. Mixing them in one queue means the slowest channel or the biggest campaign sets everyone's waiting time.
In the chapter's code, a campaign puts 2 million emails in a shared queue just before Asha asks for an OTP. At 5,000 sends a second, her OTP waits 400 seconds and expires before it arrives. With its own queue (30 OTPs ahead, 300 a second), it waits about 0.1 seconds.
Split by channel and priority:
- otp-sms (and otp-email): own workers, own provider account or sender ID, reserved provider quota.
- push / sms / email for transactional messages.
- bulk for campaigns, rate-limited so it never eats the shared provider quota.
Each queue scales its workers independently, and each has its own dead-letter queue (Chapter 22).
Interviewer pushback:
- "Doesn't that waste capacity?" → A little, in idle workers. That's the price of guaranteeing OTPs are never late, and OTP traffic is small.
Memory code: "The OTP never stands in the campaign's queue."