15. Sharding
1
The picture to keep in your head2
15.1 Why shard at all?3
15.2 How do you choose the shard key?4
15.3 What are the ways to map keys to shards?5
15.4 What is a hot shard?6
15.5 How do you add shards without moving everything?7
15.6 What happens to queries and joins that cross shards?8
15.7 How do you keep IDs and unique rules across shards?9
15.8 How do sharding and replication work together?10
Memory card and where the analogy breaks11
Practice: five levels12
Interview questions13
Solutions14
Sources15
Code lab15. Sharding
Memory card and where the analogy breaks
3 min read+10 XP
The memory card (tear this page out)
| # | Question | Memory code |
|---|---|---|
| 15.1 | Why shard? | Split the halls only when one hall can't fit the students |
| 15.2 | Choosing the shard key? | Pick the key your busiest questions already contain, with many values and even traffic |
| 15.3 | Range, hash, directory? | Ranges keep neighbours together, hashes scatter them, a list puts anyone anywhere |
| 15.4 | Hot shards? | If the key follows the clock, everyone crowds the newest hall |
| 15.5 | Resharding? | Cut the data into many small groups on day one; later, move groups, not students |
| 15.6 | Cross-shard queries? | One hall for one student; every hall for the topper, so keep topper questions rare |
| 15.7 | IDs and uniqueness? | Every hall numbers its own sheets, so stamp the hall on the number, and check uniqueness in one shared register. |
| 15.8 | With replication? | Split for size, copy for safety: every hall has its own photocopies |
The whole chapter in one line: Pick the roll number wisely, cut into many small groups from day one, keep most questions inside one hall, and photocopy every hall.
Where the analogy breaks
- Students sit still for the exam. Data keeps growing and moving, so resharding is a normal part of life, not a one-off.
- Halls don't need to agree with each other. Shards sometimes do: a transfer between two users on different shards needs both to commit or neither (Chapter 17).
- A hall's size is fixed. A shard's load isn't: one viral product or giant customer can make a shard hot overnight.
Quick check
Five questions on this chapter. Pick an answer, then check it.
quiz