Art of Codeartofcode
Lesson 2 of 16

12. Caching: The Basics

1
The picture to keep in your head
2
12.1 What is a cache, and what are a hit, a miss and the hit rate?
3
12.2 Why is a cache faster than the database?
4
12.3 Where does a cache physically live?
5
12.4 On a miss, who fetches the data?
6
12.5 When the data changes, what gets written where?
7
12.6 What happens when the real data changes, and why add jitter?
8
12.7 What happens when the cache is full?
9
12.8 How do you invalidate safely, and what's the race everyone hits?
10
12.9 When should you cache, what should you never cache, and how do you name keys?
11
Memory card and where the analogy breaks
12
Practice: five levels
13
Interview questions
14
Solutions
15
Sources
16
Code lab
12. Caching: The Basics

12.1 What is a cache, and what are a hit, a miss and the hit rate?

4 min read+10 XP

What a cache is

What is a cache?

In the kitchen: Your fridge isn't where milk comes from. Milk comes from the supermarket. The fridge just holds a copy of what you bought, kept close so you can grab it fast, and you only keep the things you use often. Nobody keeps a once-a-year ingredient in the fridge.

In the system: A cache is a separate, faster storage layer that keeps copies of data the system needs often, so it doesn't have to fetch that data from the slow original source (usually the database) every time.

  • Copies: the database keeps the original. The cache never holds the only version.
  • Faster: the whole point. A cache that isn't much faster than the source is useless.
  • Often: cache what gets asked for again and again, not what's asked for once.

Memory code: a cache is your CFO. Copies, Faster, Often.

Why do we need a cache at all?

In the kitchen: Imagine walking to the supermarket every single time you want tea. Two things go wrong. You wait 20 minutes for every cup. And if your whole building did the same, the shop would be jammed: long queues, tired staff, maybe the doors close.

In the system: It's the same two problems.

  • Users wait. Every page load pays the full trip to the database.
  • The database carries the weight. A database can only handle so many queries per second. On our grocery app, thousands of users open the milk page every minute, and the answer (name, price, photo) is identical for all of them. Without a cache, the database repeats the same work thousands of times. During a sale, that's how databases fall over.

A cache fixes both: do the work once, keep the answer, hand out copies.

Memory code: "Wait and Weight." Without a cache, users wait and the database carries all the weight.

What exactly does a cache store?

In the kitchen: Every box in a well-run fridge has a label. You don't dig through everything; you look for the box marked "milk" and grab it. The label is how you find it. The box is what you get.

In the system: A cache stores key-value pairs.

key: "product:milk" value: { "name": "Full Cream Milk 1L", "price": 52, "photo": "milk.jpg" }
  • The key is the label: a unique name.
  • The value is what's in the box: almost anything you can turn into text or bytes, like a database row, an API response, a whole rendered page, or a computed result such as "top 10 products today".

Why key-value? Looking something up by its exact name is the fastest kind of lookup. The cache jumps straight to the key instead of searching.

Memory code: "No label, no lookup." Everything in a cache is found by its exact key.

How does a request flow, step by step?

In the kitchen: You want tea. Check the fridge first. Milk's there? Make tea. Not there? Go to the shop, and on the way back, put a carton in the fridge, so next time it's there.

one request, two possible paths. A hit stops at the cache. A miss goes to the database and leaves a copy behind.

Figure: one request, two possible paths. A hit stops at the cache. A miss goes to the database and leaves a copy behind.

In the system:

1. User asks for the milk page. 2. App asks the cache: "Do you have product:milk?" 3a. YES, a HIT: the cache returns the copy. App replies. Fast. 3b. NO, a MISS: app asks the database, gets the answer, saves a copy in the cache, then replies. 4. The next user who asks for milk gets a hit.
  • Hit = the cache had the exact key you asked for and the copy hadn't expired.
  • Miss = the key wasn't there, or the copy had expired, so you went to the database.

Only the first request pays for the slow trip. Everyone after rides on it.

Memory code: "Fridge first, shop second, stock on the way back."

What is the hit rate, and why does it decide everything?

In the kitchen: If you find milk in the fridge 9 mornings out of 10, the fridge is doing its job. If it's there 2 mornings out of 10, you're still walking to the shop most days.

In the system: The hit rate is the share of lookups that were hits:

hit rate = hits ÷ (hits + misses) average time ≈ hit rate × cache time + (1 − hit rate) × database time

The misses dominate. With a 1 ms cache and a 50 ms database, a 90% hit rate averages about 5.9 ms; 99% averages about 1.5 ms, roughly four times faster, and the database gets ten times fewer queries (Interview Q2). Small improvements in hit rate near the top make a big difference.

Memory code: "Count the hits; the misses set the speed."

Is the cache the "real" data?

In the kitchen: Here the kitchen and the system differ, and the difference is the point. If your fridge breaks, you lose real milk. In a system, the "fridge" only ever holds copies. The supermarket (database) still has everything.

In the system: The database is the source of truth. The cache is a convenience copy.

  • If the cache is wiped, you lose nothing. Every request becomes a miss for a while, things slow down, and the cache refills itself.
  • Never store something only in a cache. If it isn't in the database, a restart can erase it forever.

Memory code: "Lose the copy, keep the truth."