29.10 How do health checks and failover work?

At home: The inverter watches the mains voltage. The moment it drops, it switches over, quickly enough that the fans barely slow down. But it shouldn't switch over just because the voltage dipped for a split second.

In the system:

• Health checks let load balancers and orchestrators know which instances to use. A liveness check asks "is the process alive?" (restart it if not); a readiness check asks "can it serve traffic right now?" (stop sending traffic if not). Kubernetes has both.
• Make readiness meaningful but local: check that the instance can reach what it must have (its database connection pool), but don't fail readiness because an optional dependency is down, or every instance goes unready at once and you take yourself offline.
• Detection time = check interval × failures needed: checking every 10 s and requiring 3 failures means about 30 s before traffic moves. Faster detection means more false alarms; tune both.
• Failover: moving work from a failed component to a standby: load balancers removing instances, database replicas promoted (Chapter 14), DNS or traffic managers moving users to another region (29.11).
• Avoid flapping and split brain: require several failures before acting, add hysteresis before switching back, and use fencing so an old primary can't keep writing (Chapters 14 and 17).

Memory code: "Watch the mains; switch over fast, but not at every flicker."

---