Art of Codeartofcode
Lesson 6 of 15

33. Design a Notification System

1
The interview prompt
2
33.1 What do you ask first?
3
33.2 How big is this, really?
4
33.3 What are the doors?
5
33.4 What do we store?
6
33.5 What does the whole system look like?
7
33.6 Why one queue per channel?
8
33.7 How do retries and de-duplication work?
9
33.8 How do user preferences work?
10
33.9 What makes OTPs special?
11
33.10 What would you say in the last five minutes?
12
Score yourself
13
Variations to practise
14
Sources
15
Code lab
33. Design a Notification System

33.5 What does the whole system look like?

3 min read+10 XP

Step 4: High-level design (10 minutes)

services call the notification API, which applies preferences and templates, then routes to per-channel queues; OTPs get their own fast lane; workers call providers with failover; receipts update a status store.

Figure: the final design. The green OTP lane shares nothing slow with the rest.

An order notification, step by step:

  1. The order service calls POST /v1/notifications with an idempotency key. The API de-duplicates on the key, looks up the type, the user's preferences and devices, and decides channels (33.8).
  2. It renders the template per channel and language and writes one message per channel to the channel queues (push, SMS, email), plus a status record.
  3. Channel workers take messages, apply provider rate limits, call the provider (APNs/FCM, SMS provider, email provider), and record the provider's message ID.
  4. Providers report delivery receipts by webhook (Chapter 24); the status store is updated. Failures are retried with backoff, then dead-lettered (33.7).

An OTP, step by step:

  1. Login calls POST /v1/otp/send. The OTP service checks resend limits, generates a code with a cryptographic random generator, stores its HMAC with a 5-minute expiry, and puts a message on the otp-sms queue.
  2. Dedicated OTP workers send it immediately through SMS provider A; if no success or receipt within a few seconds, fail over to provider B.
  3. The user types the code; verify checks the hash, expiry and attempt count, and deletes it on success.

Campaigns: a scheduler expands the audience in batches, applies preferences, quiet hours and caps, and feeds a separate bulk queue with its own workers and provider quota.

Interviewer pushback:

  • "Why not one queue with priorities?" → Priority queues still share workers and provider quotas; a stuck bulk job can starve everything. Separate queues and worker pools are a bulkhead (Chapter 29): the OTP lane can't be hurt by anything else.
  • "What if APNs or FCM is down?" → Push messages wait and retry; for important types, fall back to another allowed channel (email) after a delay. Push is never guaranteed anyway (Chapter 21), so important state lives in the app too.

Memory code: "One API, one queue per lane, workers that respect the provider."

The interviewer usually picks one or two. Offer: "The interesting parts are isolating channels, retries without duplicates, preferences, and the OTP itself. Which would you like?"