Art of Codeartofcode
Lesson 6 of 15

13. Caching at Scale

1
The picture to keep in your head
2
13.1 What is a cache stampede, and how do you stop one?
3
13.2 What is cache penetration?
4
13.3 What is a cache avalanche?
5
13.4 What is a hot key, and why can't sharding fix it?
6
13.5 How do you spread a cache across many machines?
7
13.6 What is cache warming?
8
13.7 How does HTTP caching work in browsers and CDNs?
9
13.8 How do you know your cache is healthy?
10
Memory card and where the analogy breaks
11
Practice: five levels
12
Interview questions
13
Solutions
14
Sources
15
Code lab
13. Caching at Scale

13.5 How do you spread a cache across many machines?

3 min read+10 XP

Running caches in production

At the wedding: One fridge isn't big enough. So you use several fridges and a rule everyone follows: dairy in fridge 1, vegetables in fridge 2, desserts in fridge 3. Anyone can find anything by the rule. And for each fridge, a spare fridge holds a copy, in case one breaks.

In the system: A distributed cache splits keys across many nodes (sharding, Chapter 15) and usually keeps a replica of each node (Chapter 14).

keys hash into 16,384 slots; three primaries each own a range; each has a replica.

Figure: a Redis-Cluster-style layout. Each key's slot decides its node; each primary has a replica.

How keys find their node:

  • Hash slots (Redis Cluster): every key maps to one of 16,384 slots (a CRC16 of the key, mod 16,384), and each primary node owns a range of slots. Adding a node means moving some slots to it; only the keys in those slots move.
  • Consistent hashing (common with Memcached clients): nodes sit on a hash ring and each key goes to the next node clockwise (Chapter 28).
  • Avoid plain hash(key) mod N: changing N from 10 to 11 nodes moves about 91% of keys, an instant avalanche. Slots or a ring move only about 1 in 11 (Chapter 7's Level 4 bug was this exact problem).

Where the routing happens: in the client library (it knows the slot map), in a proxy in front of the cluster, or in the cluster itself, which redirects clients that ask the wrong node.

Sizing (assumed): 500 GB of hot data, on 64 GB nodes kept about 75% full (headroom for spikes and fragmentation): 500 ÷ 48 ≈ 10.4, so 11 primaries, plus 11 replicas.

Two cautions:

  • Multi-key operations (fetching many keys at once, transactions) only work easily when the keys are on the same node. Redis lets you force related keys into one slot with a hash tag: only the part inside {...} is hashed, so cart:{7}:items and cart:{7}:total land together.
  • A failover promotes a replica; with asynchronous replication, the last few writes may be lost. Fine for a cache, which is a copy; never put data only there.

Memory code: "Split the stock across fridges by label, and keep a spare of each."