Art of Codeartofcode
Lesson 12 of 16

12. Caching: The Basics

1
The picture to keep in your head
2
12.1 What is a cache, and what are a hit, a miss and the hit rate?
3
12.2 Why is a cache faster than the database?
4
12.3 Where does a cache physically live?
5
12.4 On a miss, who fetches the data?
6
12.5 When the data changes, what gets written where?
7
12.6 What happens when the real data changes, and why add jitter?
8
12.7 What happens when the cache is full?
9
12.8 How do you invalidate safely, and what's the race everyone hits?
10
12.9 When should you cache, what should you never cache, and how do you name keys?
11
Memory card and where the analogy breaks
12
Practice: five levels
13
Interview questions
14
Solutions
15
Sources
16
Code lab
12. Caching: The Basics

Practice: five levels

3 min read+10 XP

Assignments

Five levels. Do them in order. Answers are in "Solutions" at the end of the chapter, so try first.

Level 1: Think (no code)

For each item, decide: cache it or not? If yes, suggest a TTL and say why. Use "Cache what can REST, skip the three Us."

  1. The list of countries in a sign-up form dropdown.
  2. A user's bank balance on the "confirm payment" screen.
  3. The "trending now" list on a shopping app's home page.
  4. Search results for whatever random text a user types.
  5. A user's profile photo URL, shown next to every comment they post.
  6. The "likes" count on product reviews, updated thousands of times a minute. Which write strategy fits, and why?

Level 2: Calculate

Your app gets 1,000 requests per second for product pages.

  • A cache read takes 2 ms. A database read takes 40 ms.
  • Your cache has an 80% hit rate. (Ignore the small cost of checking the cache on a miss.)
  1. What's the average response time?
  2. How many queries per second reach the database?
  3. Your database starts struggling above 100 queries per second. What hit rate do you need to stay under that?
  4. After a deploy, 10,000 product keys are refilled within the same second, all with a 300-second TTL. What happens 300 seconds later? What TTL range would spread those expiries over about a minute?

Level 3: Build

Write a small in-memory cache class, TTLCache(capacity, ttl_seconds), with:

  • get(key): returns the value, or None if the key is missing or expired.
  • set(key, value): stores the value with an expiry time.
  • delete(key): removes the key.
  • LRU eviction: when there are more than capacity items, remove the least recently used one.
  • hit_rate(): hits ÷ (hits + misses).

Then use it in front of a fake slow database (get_price, update_price) and check that:

  1. Five reads of the same key call the database only once.
  2. After update_price, the next read returns the new price.
  3. With capacity=2, reading a third key evicts the least recently used one.
  4. After the TTL passes, a read goes back to the database.

Extension: add (a) a WriteBack wrapper whose writes reach the database only on flush(), and show that a crash before the flush loses the write; and (b) TTL jitter, showing that 10,000 keys no longer expire in the same second.

Hint: Python's collections.OrderedDict has move_to_end() and popitem(last=False). Pass in a fake clock so tests don't have to sleep.

Level 4: Debug

Users complain: "After the admin changes a price, some of us still see the old price for up to 10 minutes." Here's the code:

python
  1. Find the bug. Explain step by step how a user ends up seeing the old price for 10 minutes.
  2. Fix it.

Level 5: Design

You're designing the product page for an online store's biggest sale of the year.

  • 5 million users open product pages in the first 10 minutes.
  • 200 products are on sale. 10 of them get most of the traffic.
  • Prices can change during the sale (flash discounts), and a user must never pay a wrong price.
  • Stock changes every second.

Deliver:

  1. What do you cache, and what do you never cache?
  2. Which cache layers do you use (browser, CDN, Redis, in-app)?
  3. What TTL for each kind of data, and why?
  4. How do you make sure checkout never uses a stale price?
  5. What breaks first if the cache goes down, and how do you protect the database?