Art of Codeartofcode
Lesson 11 of 15

13. Caching at Scale

1
The picture to keep in your head
2
13.1 What is a cache stampede, and how do you stop one?
3
13.2 What is cache penetration?
4
13.3 What is a cache avalanche?
5
13.4 What is a hot key, and why can't sharding fix it?
6
13.5 How do you spread a cache across many machines?
7
13.6 What is cache warming?
8
13.7 How does HTTP caching work in browsers and CDNs?
9
13.8 How do you know your cache is healthy?
10
Memory card and where the analogy breaks
11
Practice: five levels
12
Interview questions
13
Solutions
14
Sources
15
Code lab
13. Caching at Scale

Practice: five levels

3 min read+10 XP

Assignments

Five levels. Do them in order. Answers are in "Solutions" at the end, so try first.

Level 1: Think (no code)

Name the problem (stampede, penetration, avalanche, hot key) and one fix.

  1. After a midnight deploy restarts all app servers and empties the cache, the database CPU hits 100% for 10 minutes.
  2. A scraper requests /product/1, /product/2... up to /product/50000000, though only 200,000 products exist.
  3. The home-page banner key expires every 5 minutes, and each time the database shows a short spike of identical queries.
  4. During a cricket final, one cache node is at 100% CPU and the others at 20%.
  5. Every product was cached by a nightly job at 2:00 am with a 24-hour TTL. Every night at 2:00 am, the database slows down.

Level 2: Calculate

  1. The milk page gets 5,000 requests a second; rebuilding it takes 200 ms. Without protection, about how many database queries hit at once when it expires?
  2. 1 million keys were cached within the same 10 minutes with the same 1-hour TTL. How many extra misses a second will there be when they expire? With a ±30-minute jitter instead?
  3. You need 500 GB of hot data in cache. Nodes have 64 GB; keep them 75% full. How many primaries? With one replica each?
  4. You grow the cluster from 10 to 11 nodes. Roughly what fraction of keys move with hash mod N? With slots or consistent hashing?

Level 3: Build

In Python (standard library only), write:

  1. A SingleFlightCache where, if 100 threads miss the same key at once, the database is called once and everyone gets the value.
  2. A NegativeCache that caches "not found".
  3. The CRC16 function used by Redis Cluster, and slot(key) = crc16(key) % 16384. Check it against the known value crc16(b"123456789") == 0x31C3, and that 40,000 keys spread roughly evenly over 4 nodes.
  4. A serve(body, if_none_match) that returns 200 with an ETag, or 304 with no body when the ETag matches.

Level 4: Debug

The team added stampede protection, but the database still spikes when the banner key expires.

python
  1. Find the bug (there are two problems; find both). Explain what happens with 100 concurrent misses.
  2. Fix it.

Level 5: Design

The grocery app's Big Sale starts at 8 pm. Expect 50,000 requests a second at peak; 200 products are on sale; one product will be on the home-page banner; bots are known to scrape product IDs.

Deliver:

  1. Your cache layers and what each holds.
  2. How you prevent a stampede on the banner product and an avalanche at 8 pm.
  3. How you handle the hot banner key.
  4. How you stop bots from penetrating to the database.
  5. The cluster layout and what happens if one cache node dies at 8:05 pm.
  6. Three alerts you'd set.