Art of Codeartofcode
Lesson 7 of 16

12. Caching: The Basics

1
The picture to keep in your head
2
12.1 What is a cache, and what are a hit, a miss and the hit rate?
3
12.2 Why is a cache faster than the database?
4
12.3 Where does a cache physically live?
5
12.4 On a miss, who fetches the data?
6
12.5 When the data changes, what gets written where?
7
12.6 What happens when the real data changes, and why add jitter?
8
12.7 What happens when the cache is full?
9
12.8 How do you invalidate safely, and what's the race everyone hits?
10
12.9 When should you cache, what should you never cache, and how do you name keys?
11
Memory card and where the analogy breaks
12
Practice: five levels
13
Interview questions
14
Solutions
15
Sources
16
Code lab
12. Caching: The Basics

12.6 What happens when the real data changes, and why add jitter?

3 min read+10 XP

Keeping it fresh and small

TTL and delete-on-change

In the kitchen: Your fridge has Monday's milk. On Tuesday the shop gets a new batch at a new price. Your fridge doesn't know. It's out of date. You fix that two ways: milk has a use-by date, after which you don't trust it, and if you hear the shop changed something, you throw out the old carton right away.

In the system: Milk's price changes from ₹52 to ₹56 in the database. The cache still says ₹52. The cache is now stale. Two tools:

  • TTL ("time to live") = the use-by date. Every copy is saved with a timer, say 60 seconds. After that it's treated as gone, and the next request fetches the fresh price. Simple, but users may see ₹52 until the timer runs out.
  • Invalidation = throwing out the old carton. When the price changes, the app also deletes the product:milk key from the cache. The next request is a miss and loads ₹56 straight away.

Most systems use both: delete on change, with a timer as a safety net in case a delete gets missed.

with only a TTL, users see the old price until the timer runs out. Deleting on change closes that window.

Figure: with only a TTL, users see the old price until the timer runs out. Deleting on change closes that window.

Memory code: "Date it and delete it." Put a use-by date (TTL) on every copy, and delete it when the truth changes.

Jitter

In the kitchen: If you bought fifty cartons on the same morning with the same use-by date, they all go off on the same day, and you're back at the shop buying fifty at once.

In the system: If many keys are cached at the same moment with the same TTL (after a deploy, a cache restart, or a nightly warm-up), they all expire in the same second, and every one of them misses at once. That burst of database reads can knock the database over (Chapter 13 calls this an avalanche).

Jitter adds a small random amount to each TTL so expiries spread out:

python

10,000 keys with the same TTL expire in one second; with jitter they spread over about a minute.

Figure: in the chapter's code, 10,000 keys with a fixed TTL all expire in the same second; with ±30 s of jitter, the busiest second has 208.

Memory code: "Don't let every carton go off on the same morning."