13. Caching at Scale
The picture to keep in your head
Every question below is answered twice: first in the kitchen, so you can picture it, then in the system, so you know exactly how it works. Each answer ends with a memory code to lock it in.
The picture to keep in your head
Chapter 12's kitchen was a family kitchen: one fridge, a few people making tea. Now the same kitchen is feeding a wedding. Five hundred guests, all hungry at 1 pm. When the fridge runs out of milk, everyone runs to the shop at once. Guests keep asking for a dish that doesn't exist, and someone goes to the shop each time to check. Every carton was bought on the same day, so they all go off together. Everyone wants the biryani, so one pot has a queue out the door. One fridge isn't enough any more, so you need many fridges, and a plan for when one breaks.
At scale, the cache stops being a convenience. It becomes the thing holding the database up. This chapter is about what goes wrong when it fails, and how big systems run it.
Our running example: the grocery app from Chapter 12 during its Big Sale. The milk page that a few thousand people viewed per minute now gets thousands of views per second.
| At the wedding | In the system |
|---|---|
| Fridge empties and everyone runs to the shop | Cache stampede |
| Guests keep asking for a dish that doesn't exist | Cache penetration |
| All the milk goes off on the same day, or the fridge breaks | Cache avalanche |
| Everyone queues at the one biryani pot | Hot key |
| Many fridges, each holding part of the stock | Distributed cache (a cluster) |
| Stocking the fridge before the guests arrive | Cache warming |
| Guests keeping a plate at their own table | Browser and CDN (HTTP) caching |
Figure: at a wedding, the fridge stops being a convenience. If it fails, the shop is flattened.
This chapter builds on Chapter 12's memory codes ("Fridge first...", "Date it and delete it", "Pocket or Pantry"); have them handy.