14.7 What happens when the leader fails?

When the leader dies

At the bank: Head office goes silent. Someone has to notice, pick the branch with the most up-to-date copies, announce "this branch is now head office", and make sure every courier now goes there.

In the system: Failover turns a follower into the new leader. Steps, whether automatic or by hand:

1. Detect the failure: the leader stops answering health checks or heartbeats for some timeout. Too short a timeout causes needless failovers during a brief slowdown; too long means a longer outage.
2. Choose a new leader: usually the follower that's most up to date, to lose the least data.
3. Promote it, and reconfigure everything: the other followers now replicate from it, and apps send writes to it (by updating a DNS name, a virtual IP, or a proxy's configuration).
4. Deal with the old leader: if it comes back, it must not think it's still in charge (14.8).

What can go wrong:

• With asynchronous replication, writes the old leader confirmed but never shipped are lost (14.3).
• If those lost rows had IDs that the new leader reuses, other systems (a cache, an email already sent) may now point at the wrong data.
• Failover takes time: typically seconds to a minute or two. Apps need retries with backoff (Chapter 29).

Tools like Patroni (PostgreSQL) or Orchestrator (MySQL), and managed cloud databases, automate this.

Memory code: "Notice, choose the freshest, promote, redirect, and silence the old boss."

---