7.4 How does the load balancer know a server is dead?

At the airport: Every few minutes the queue manager glances at each counter. If an agent hasn't answered three times in a row, they put up a "closed" sign. When the counter works again for a couple of checks, they take the sign down.

In the system: Health checks.

• Active checks: the load balancer calls an endpoint like GET /health on every server every few seconds. If a server fails a set number of checks in a row (the unhealthy threshold), it's taken out of rotation. When it passes a set number in a row (the healthy threshold), it's put back.
• Passive checks: the load balancer also watches real traffic. A server returning errors or timing out on live requests gets marked down.

Figure: with checks every 10 seconds and 3 failures needed, about 25 seconds of traffic still goes to a dead server.

Designing the health endpoint well:

• Check what matters, cheaply. "The process is running" isn't enough if the server can't reach its database. But a health check that runs a heavy query on every call can itself overload the database.
• Don't fail everything together. If every server's health check calls the same database, a short database blip can mark every server unhealthy at once, turning a small problem into a full outage. Many teams separate a liveness check ("restart me if this fails") from a readiness check ("don't send me traffic right now").
• Tune the timing. Faster checks and lower thresholds catch failures sooner, but a server that had one slow moment may be pulled out unfairly.

Connection draining: when a server is being removed on purpose (scaling in, deploying), the load balancer stops sending it new requests but lets in-flight ones finish, up to a timeout. Without draining, every deploy cuts off users mid-request.

Memory code: "Three misses and the sign goes up; two passes and it comes down." And drain before you remove.

---