15. Sharding
The picture to keep in your head
Every question below is answered twice: first at the exam centre, so you can picture it, then in the system, so you know exactly how it works. Each answer ends with a memory code to lock it in.
The picture to keep in your head
Ten thousand students arrive for an exam. No hall in the city can seat them all. So the exam centre splits them across several halls, using one simple rule printed on a notice board: "Roll numbers 1 to 2,500: Hall 1. 2,501 to 5,000: Hall 2..." Every student goes to exactly one hall, and anyone can work out which one from their roll number. Each hall has its own invigilators and runs on its own.
It works beautifully, until the rule is bad. If the rule were "by date of registration", every late registrant would pile into the last hall. And a question like "which student scored highest overall?" now needs answers from every hall.
That's sharding: splitting one big dataset across several databases, each holding a slice.
Our running example: the bookstore's orders. Replication (Chapter 14) gave it copies for safety and reads, but every copy still holds all orders and applies every write. As the bookstore grows to billions of orders and tens of thousands of writes a second at sale time, one leader can't hold or write them all.
| At the exam centre | In the system |
|---|---|
| Too many students for one hall | Too much data, or too many writes, for one database |
| A hall | A shard (one database holding a slice of the data) |
| Roll number | The shard key (partition key) |
| "Rolls 1 to 2,500: Hall 1" | Range sharding |
| A formula on the roll number | Hash sharding |
| A printed list of who sits where | Directory (lookup-table) sharding |
| One hall overflowing while others are empty | A hot shard |
| Opening a new hall mid-year | Resharding |
| "Top scorer overall" | A cross-shard (scatter-gather) query |
Figure: every student goes to exactly one hall, decided by one rule everyone can apply.