Art of Codeartofcode
Lesson 5 of 15

13. Caching at Scale

1
The picture to keep in your head
2
13.1 What is a cache stampede, and how do you stop one?
3
13.2 What is cache penetration?
4
13.3 What is a cache avalanche?
5
13.4 What is a hot key, and why can't sharding fix it?
6
13.5 How do you spread a cache across many machines?
7
13.6 What is cache warming?
8
13.7 How does HTTP caching work in browsers and CDNs?
9
13.8 How do you know your cache is healthy?
10
Memory card and where the analogy breaks
11
Practice: five levels
12
Interview questions
13
Solutions
14
Sources
15
Code lab
13. Caching at Scale

13.4 What is a hot key, and why can't sharding fix it?

3 min read+10 XP

At the wedding: There are ten serving stations, but everyone wants the biryani, and there's only one biryani pot at one station. That station has a queue out the door while the other nine are idle. Adding more stations doesn't help, because the biryani is still in one pot.

In the system: In a cache cluster (13.5), each key lives on one node. A hot key is a key that gets a huge share of traffic: the product on the sale banner, a celebrity's profile, the live score of a cricket final. All its reads land on one node, which saturates its CPU or network while the other nodes sit idle.

one viral product overloads one node; copies spread it across nodes plus a small in-app cache.

Figure: sharding spreads different keys; it can't spread one key. Copies and a pocket cache can.

Fixes:

  1. A small in-app ("pocket") cache in front of the cluster for the top few hundred keys, with a short TTL (a few seconds). Most hot reads never leave the app server (Chapter 12's "Pocket or Pantry").
  2. Copies of the key: store the value under several keys (product:milk#0 ... product:milk#7), which hash to different nodes, and have each read pick one at random. Writes must update or delete all copies. In the chapter's code, one key always lands on one node; 8 copies spread across all 4 nodes.
  3. Read replicas of the hot node, with reads spread across them.

Finding hot keys: watch per-key request counts in your app, sample traffic, or use the cache's own tools (Redis, for example, can report its hottest keys when using an LFU eviction policy).

Memory code: "One pot can't feed the whole wedding: make more pots."