1Show the Level 1 solution
21. **Avalanche** (the whole cache emptied). Fix: warm the hot keys before taking full traffic, keep cache replicas or persistence, and limit database load during refill.
32. **Penetration.** Fix: a Bloom filter of valid product IDs, negative caching, ID validation and rate limiting.
43. **Stampede** on the banner key. Fix: single flight, or serve stale while one request refreshes, or refresh early.
54. **Hot key** on that node. Fix: a short-TTL in-app cache for the score, or copies of the key on several nodes.
65. **Avalanche from mass expiry.** Fix: add jitter to the TTLs (for example 24 hours ± 2 hours).
Show the Level 2 solution
1. About 5,000 × 0.2 = **1,000** identical queries before the first refill completes.
2. 1,000,000 ÷ 600 s ≈ **1,700 extra misses a second** for 10 minutes. Spread over an hour of jitter: 1,000,000 ÷ 3,600 ≈ **280 a second**, and spread out instead of concentrated.
3. Usable per node: 64 × 0.75 = 48 GB. 500 ÷ 48 ≈ 10.4 → **11 primaries**; with one replica each, **22 nodes**.
4. With `hash mod N`, about **10/11 ≈ 91%** of keys move. With slots or consistent hashing, only about **1/11 ≈ 9%**.
1Show the Level 3 solution
2Tested solution (Python 3), the key part:
34```python
5class SingleFlightCache:
6 def __init__(self, db):
7 self.db, self.store = db, {}
8 self.lock = threading.Lock()
9 self.inflight = {} # key -> Event the waiters block on
1011 def get(self, key):
12 if key in self.store:
13 return self.store[key]
14 with self.lock:
15 if key in self.store: # filled while we waited for the lock
16 return self.store[key]
17 event = self.inflight.get(key)
18 leader = event is None
19 if leader:
20 event = self.inflight[key] = threading.Event()
21 if leader:
22 try:
23 self.store[key] = self.db.get(key)
24 finally:
25 with self.lock:
26 del self.inflight[key]
27 event.set() # wake everyone, even if the load failed
28 return self.store[key]
29 event.wait(timeout=2)
30 return self.store.get(key) # use the leader's result
31```
3233All checks pass: 100 concurrent misses cause **1** database call; 1,000 lookups of a missing product cause **1** with negative caching; `crc16(b"123456789") == 0x31C3`; 40,000 keys land roughly evenly on 4 nodes; and a matching ETag returns `304` with no body. The full file is in `code/ch13_cache_at_scale.py`.
1Show the Level 4 solution
2**Problem 1 (the main bug):** after waiting, followers call **`self.db.get(key)`** instead of reading the value the leader stored. The leader does load once, but then every one of the 99 waiters also queries the database. In the tested demo, 100 concurrent misses still cause **100** database calls. The protection only delays the stampede.
34**Problem 2:** if `self.db.get` raises an error in the leader, `inflight[key]` is **never deleted** and the event is **never set**. Every later request for that key finds an "in-flight" entry and waits for the full timeout, then (with problem 1) hits the database anyway. One failed load slows that key for everyone.
56**The fix:**
78```python
9 if leader:
10 try:
11 self.store[key] = self.db.get(key)
12 finally: # always clean up and wake the waiters
13 with self.lock:
14 del self.inflight[key]
15 event.set()
16 return self.store[key]
17 event.wait(timeout=2)
18 return self.store.get(key) # read the leader's result, not the database
19```
2021Also re-check the store after taking the lock, so a request that arrives just after the leader finishes doesn't become a new leader.
1Show the Level 5 (sketch) solution
21. **Layers:** CDN for images, scripts and short-lived product page HTML (`s-maxage` of 30 to 60 seconds, purged on price change); Redis cluster for product data and display prices; a small in-app cache (5 to 10 seconds) for the banner product and the top 100 products; prices at checkout always from the database.
32. **Stampede and avalanche:** single flight (distributed lock with expiry) on every product key; serve stale while revalidating; **warm** the 200 sale products and the home page at 7:45 pm with **jittered TTLs** (for example 300 ± 60 s); never let all sale keys share one expiry.
43. **Hot banner key:** in-app cache in front, plus 8 copies of the key across nodes; updates delete all copies.
54. **Bots:** Bloom filter of valid product IDs, negative caching for 60 seconds, ID format validation, and rate limits per client on product lookups.
65. **Cluster:** hash slots across, say, 6 primaries with one replica each, in different zones. If a node dies at 8:05, its replica is promoted within seconds; a few recent writes may be lost (acceptable for a cache); during failover, timeouts and database limits stop the database being flooded.
76. **Alerts:** hit rate drops more than 20 points in 5 minutes; any node's load more than 3 times the average; database queries a second above a set threshold while cache errors rise.