15. Sharding
1
The picture to keep in your head2
15.1 Why shard at all?3
15.2 How do you choose the shard key?4
15.3 What are the ways to map keys to shards?5
15.4 What is a hot shard?6
15.5 How do you add shards without moving everything?7
15.6 What happens to queries and joins that cross shards?8
15.7 How do you keep IDs and unique rules across shards?9
15.8 How do sharding and replication work together?10
Memory card and where the analogy breaks11
Practice: five levels12
Interview questions13
Solutions14
Sources15
Code lab15. Sharding
15.4 What is a hot shard?
3 min read+10 XP
What goes wrong
At the exam centre: Registration opened in January. If halls are assigned by registration date, every student who registered this week, which is most of them, crowds into the newest hall, while the January hall sits nearly empty.
In the system: A hot shard gets far more traffic than the others, so it saturates while the rest idle. Common causes:
- A key that grows with time under range sharding: all new writes go to the newest shard. In the chapter's code, sharding orders by day sends all 10,000 of today's orders to one shard; hashing by user spreads them at about 2,500 each.
- A few huge keys: one celebrity, one giant tenant, one viral product, all on one shard.
- Low-cardinality keys: sharding by country or status gives only a few, very uneven buckets.
Figure: a key that grows with time makes one shard do all the work.
Fixes:
- A better key: hash a key with many values that spreads writes (user ID instead of date).
- Compound keys: for time-series data, shard by
(device_id, day)or add a small random bucket (order_day + bucket 0..15), and query all buckets for a day. - Split the hot range or isolate the big tenant (directory sharding).
- Cache the hot reads (Chapter 13's hot keys) so the shard sees fewer of them.
Memory code: "If the key follows the clock, everyone crowds the newest hall."