Art of Codeartofcode
Lesson 9 of 17

14. Replication

1
The picture to keep in your head
2
14.1 Why keep copies of the same data?
3
14.2 How does leader-follower replication work?
4
14.3 Should the leader wait for followers before confirming a write?
5
14.4 What is replication lag, and why does it happen?
6
14.5 How do you stop lag from confusing users?
7
14.6 How do you use read replicas in practice?
8
14.7 What happens when the leader fails?
9
14.8 What is split brain, and how is a leader safely chosen?
10
14.9 When would you have more than one leader?
11
14.10 How does leaderless replication work, and what are quorums?
12
Memory card and where the analogy breaks
13
Practice: five levels
14
Interview questions
15
Solutions
16
Sources
17
Code lab
14. Replication

14.8 What is split brain, and how is a leader safely chosen?

3 min read+10 XP

At the bank: The phone line between head office and the branches goes dead. The branches assume head office is gone and appoint a new one. But the old head office is fine, just cut off, and keeps issuing notices to the customers in front of it. Now there are two head offices issuing contradicting orders.

In the system: Split brain is when two nodes both believe they're the leader and both accept writes. Data diverges, and someone must later merge or discard writes by hand.

a network cut leaves the old leader and the promoted follower both accepting writes for the same seat.

Figure: the old leader wasn't dead, just unreachable. Both sides sold seat 14A.

A real example: in October 2018, a brief network partition at GitHub led its automated failover to promote MySQL primaries in another region, while writes had already landed in the original one. GitHub's post-incident analysis describes about a day of degraded service while engineers reconciled the data, and the change they made afterwards: no automatic promotion across regions.

How systems prevent it:

  1. Majority (quorum) elections: a node becomes leader only with votes from a majority of the cluster. A side cut off in the minority can't elect anyone, so there can be at most one leader. Raft and Paxos are the well-known algorithms; tools like etcd, ZooKeeper and Consul provide this so databases don't each reinvent it. This is why clusters use odd numbers (3 or 5): 3 nodes survive 1 failure, 5 survive 2.
  2. Fencing: every new leader gets a higher term (or epoch) number. Storage and other nodes reject writes carrying an old term, so a returning old leader is ignored. Some setups go further and forcibly cut the old leader off before promoting (sometimes called "shoot the other node in the head").
  3. Leader leases: a leader only acts while it holds a lease that expires; the cluster waits for the old lease to run out before electing a new leader.

Memory code: "Only a majority can crown a leader, and the old crown must stop working."