Art of Codeartofcode
Lesson 13 of 17

14. Replication

1
The picture to keep in your head
2
14.1 Why keep copies of the same data?
3
14.2 How does leader-follower replication work?
4
14.3 Should the leader wait for followers before confirming a write?
5
14.4 What is replication lag, and why does it happen?
6
14.5 How do you stop lag from confusing users?
7
14.6 How do you use read replicas in practice?
8
14.7 What happens when the leader fails?
9
14.8 What is split brain, and how is a leader safely chosen?
10
14.9 When would you have more than one leader?
11
14.10 How does leaderless replication work, and what are quorums?
12
Memory card and where the analogy breaks
13
Practice: five levels
14
Interview questions
15
Solutions
16
Sources
17
Code lab
14. Replication

Practice: five levels

3 min read+10 XP

Assignments

Five levels. Do them in order. Answers are in "Solutions" at the end, so try first.

Level 1: Think (no code)

A. Which problem is this, and what's the fix?

  1. A user posts a review, refreshes the page, and their review is missing; a second later it appears.
  2. Scrolling through order history, a user sees an order, then on the next page load it's gone, then it's back.
  3. After the database leader crashed and a replica took over, three customers who were told "order confirmed" have no order.
  4. During a network glitch, two database nodes both accepted writes; now some seats are double-booked.

B. Synchronous or asynchronous replication, and why?

  1. The bookstore's payments table.
  2. Page-view counters.
  3. A replica in another continent used for reports.

Level 2: Calculate

  1. The leader takes 1,000 writes a second; a follower applies 900. How far behind is it after 10 minutes? If writes drop to 500 a second, how long until it catches up?
  2. Replication is asynchronous and a follower is typically 2 seconds behind at 1,000 writes a second. Roughly how many confirmed writes could be lost in a failover?
  3. Each replica handles 5,000 reads a second at full load; you run them at 70%. How many replicas for 20,000 reads a second?
  4. With N = 5: which W and R give W + R > N with both reads and writes surviving 2 nodes down?

Level 3: Build

In Python:

  1. A Leader with a list of (key, value) log entries and write() that returns the entry's log position, and a replicate(follower, max_entries) that ships entries asynchronously.
  2. read_your_writes(user_last_lsn, leader, follower, key) that reads from the follower only if it has applied the user's last write.
  3. quorum_write(replicas, key, value, version, w) and quorum_read(replicas, key, r) that returns the highest version.

Check that: a read on a lagging follower is stale but read-your-writes returns the new value from the leader; once the follower catches up, it serves the read; promoting a follower that applied 7 of 10 writes loses 3; with N = 3, W = 2, R = 2 always returns the latest; R = 1 can be stale.

Level 4: Debug

Users report: "I changed my delivery address, the page still shows the old one, so I changed it again. Now I have two support tickets."

python
  1. Find the bug. Explain step by step.
  2. Fix it. Give two fixes.

Level 5: Design

The bookstore's database must survive a machine failure with no lost orders, handle 4 times today's browsing reads, and serve users in India and Europe.

Deliver:

  1. The replication layout (leader, followers, regions) and sync vs async choices.
  2. How reads are routed, including read-your-writes for carts and orders.
  3. How failover works, and how you avoid split brain.
  4. What happens to European users' writes, and why.
  5. What you monitor.