14.4 What is replication lag, and why does it happen?

Living with lag

At the bank: The courier to a far branch takes hours. On a busy day with hundreds of notices, the courier falls behind, and that branch's copies get older and older.

In the system: Replication lag is how far a follower is behind the leader, measured in time ("2 seconds behind") or in log position. With asynchronous replication it's normally milliseconds, but it grows when:

• the follower is slower or busier than the leader (heavy read queries on it, a smaller machine);
• a huge write happens (bulk updates, a migration, a big delete);
• the network between them is slow, as with a replica in another region.

If writes arrive faster than a follower can apply them, the lag grows without limit. Example (assumed): the leader takes 1,000 writes a second and a follower can apply 900. After 10 minutes, it's 60,000 changes behind. When writes drop to 500 a second, it catches up at 400 a second, which takes 150 seconds.

Watch it: databases report lag (PostgreSQL's pg_stat_replication view shows how far each standby has written, flushed and replayed). Alert when it grows, and stop sending reads to a follower that's too far behind.

Memory code: "The courier can fall behind; always know by how much."

---