1Show the Level 1 solution
2**A.**
31. **Sync**: the order depends on the payment result.
42. **Async.**
53. **Async**: search can be a few seconds behind (Chapter 19).
64. **Sync**: you need the answer before accepting the order.
75. **Async**: a background job.
89**B.**
101. **At-most-once** lost the message; ack after processing instead.
112. **At-least-once duplicate**; make the SMS consumer idempotent and pass the order ID as an idempotency key to the gateway.
123. **Poison message**; cap retries and dead-letter it.
134. **Ordering**; partition by order ID, or check state and versions.
145. **Kafka's partition limit**: one consumer per partition per group, so the 7th sits idle. Add partitions.
1Show the Level 2 solution
21. Sync: 60 + 800 + 600 + 300 = **1,760 ms**. Async: about **60 ms** (save and enqueue).
32. (1,200 − 1,000) × 3,600 = **720,000** messages. Drain at 1,000 − 400 = 600 a second: 720,000 ÷ 600 = **1,200 s = 20 minutes**.
43. 1,000 × 0.2 = **200** workers; with 50% headroom, **300**.
54. 2 + 4 + 8 + 16 + 32 = **62 seconds**.
65. At most **12** consumers per group (more sit idle). Two services, **two consumer groups**, each reading every message.
1Show the Level 3 solution
2Tested solution (Python 3), the key parts:
34```python
5class Queue:
6 def receive(self, now):
7 for mid in list(self.order):
8 if mid not in self.messages:
9 continue
10 if self.invisible_until.get(mid, -1) > now:
11 continue # someone is working on it
12 if self.receives[mid] >= self.max_receives:
13 self.dlq.append(self.messages.pop(mid)) # poison: to the DLQ
14 continue
15 self.receives[mid] += 1
16 self.invisible_until[mid] = now + self.visibility_timeout
17 return mid, self.messages[mid]
18 return None
1920 def ack(self, mid):
21 self.messages.pop(mid, None)
222324class Loyalty:
25 def award_idempotent(self, order_id, user, pts):
26 if order_id in self.applied: # real life: UNIQUE constraint
27 return False
28 self.applied.add(order_id)
29 self.points[user] += pts
30 return True
313233class Log:
34 def poll(self, group, max_n=10):
35 start = self.offsets[group]
36 batch = self.entries[start:start + max_n]
37 self.offsets[group] = start + len(batch)
38 return batch
3940 def seek(self, group, offset):
41 self.offsets[group] = offset
42```
4344All checks pass: an un-acked message returns after 30 s; a poison message reaches the DLQ after 3 tries; at-most-once processes 98 of 100 with two crashes and at-least-once 102; the idempotent consumer gives 50 points, not 100; partitioning keeps every order's steps in order while 8 free workers scramble them; the log serves two groups and replays; the scheduler releases jobs on time. The full file is in `code/ch22_queues.py`.
1Show the Level 4 solution
2**What happens:**
31. Worker A receives the message at second 0. It's invisible until second 30.
42. The points service is slow; A is still waiting at second 30.
53. At second 30 the timeout expires and the message becomes **visible again**. Worker B receives it at second 31.
64. Both A and B call `points_service.add(asha, 50)`. Asha gets **100** points. The tested demo shows exactly this.
78**Fix 1, the timeout:** set the visibility timeout well above the slowest normal processing time (say 120 s), or extend it while working:
910```python
11msg = queue.receive()
12with keep_extending_visibility(msg, every=20, by=60): # heartbeat while busy
13 process(msg)
14queue.ack(msg)
15```
1617**Fix 2, idempotency (the one that matters):** duplicates also come from crashes and producer retries, so make the effect happen once whatever the cause:
1819```python
20def points_worker():
21 while True:
22 msg = queue.receive()
23 order_id, user, pts = msg.body
24 with db.transaction():
25 inserted = db.execute(
26 "INSERT INTO points_awarded (order_id) VALUES (?) ON CONFLICT DO NOTHING",
27 (order_id,)).rowcount
28 if inserted:
29 db.execute("UPDATE wallets SET points = points + ? WHERE user = ?", (pts, user))
30 queue.ack(msg)
31```
1Show the Level 5 (sketch) solution
21. **Topology:** checkout writes the order and an **outbox** row in one transaction; a relay publishes "order placed" to a **Kafka topic** keyed by order ID. Consumer groups: **SMS**, **store printing**, **loyalty**, **analytics**, each reading every event. (Equivalent with SNS fanning out to one SQS queue per consumer, Chapter 23.)
32. **Guarantees:** all at-least-once. SMS: de-duplicate on (order ID, "confirm") and pass it as the gateway's idempotency key. Printing: a printed-slips table keyed by order ID. Loyalty: the unique-constraint pattern from Level 4. Analytics: tolerates rare duplicates or de-duplicates downstream.
43. **Big Sale:** 1,200 orders a second against an SMS limit of 500 a second. Cap the SMS consumers' total send rate at about 450 a second (a token bucket, Chapter 25); the backlog grows by about 750 a second, so after an hour about 2.7 million SMS are waiting, and they drain afterwards. If that delay is unacceptable, send the confirmation by push notification first, or negotiate a higher SMS limit. Printing and loyalty scale out normally.
54. **Ordering:** keying by order ID keeps each order's events in sequence (placed, then cancelled, for example). No global ordering needed.
65. **Failures:** retries with exponential backoff and jitter (2 s up to 5 minutes); after 5 failures, a per-consumer DLQ topic or queue; alerts on DLQ size, consumer lag and the oldest message's age; a redrive tool to replay fixed messages.