15. Sharding
1
The picture to keep in your head2
15.1 Why shard at all?3
15.2 How do you choose the shard key?4
15.3 What are the ways to map keys to shards?5
15.4 What is a hot shard?6
15.5 How do you add shards without moving everything?7
15.6 What happens to queries and joins that cross shards?8
15.7 How do you keep IDs and unique rules across shards?9
15.8 How do sharding and replication work together?10
Memory card and where the analogy breaks11
Practice: five levels12
Interview questions13
Solutions14
Sources15
Code lab15. Sharding
15.7 How do you keep IDs and unique rules across shards?
3 min read+10 XP
At the exam centre: Each hall numbering answer sheets 1, 2, 3 gives you twenty "sheet 57"s. And the rule "no two students share a seat number" is easy inside one hall, but who checks it across halls?
In the system: Two things a single database gave you for free now need design:
- Unique IDs: each shard's auto-increment would repeat. Use globally unique IDs: Snowflake-style IDs, UUID v7, or ID ranges per shard (Chapter 10). Putting the shard number inside the ID (as Instagram's scheme does) lets any service find a row's shard from its ID alone.
- Unique constraints on a column that isn't the shard key, like "one account per email" when users are sharded by user ID. Two sign-ups with the same email could land on different shards, each seeing no conflict. Fixes:
- A separate uniqueness table keyed by the unique value (
email → user_id), itself sharded by email. Sign-up first claims the email there (an insert that fails if it exists), then creates the user. If the second step fails, release the claim. - Make that column the shard key, if it fits the queries.
- A dedicated uniqueness service.
- A separate uniqueness table keyed by the unique value (
- Transactions across shards (moving money between two users on different shards) need two-phase commit or a saga (Chapter 17). Design so most transactions stay inside one shard.
Memory code: "Every hall numbers its own sheets, so stamp the hall on the number, and check uniqueness in one shared register."