21.4 How do heartbeats, reconnects and missed messages work?

At the gate: On a long phone call, if it goes quiet, you ask "are you still there?" If the line drops, you call back, but not all at the same instant as every other caller. And when you get through, you ask "where were we?", so you don't miss what was said while you were cut off.

In the system: Long-lived connections will break: phones switch from Wi-Fi to mobile data, go through tunnels, sleep; servers restart. Three things make that safe.

Heartbeats. A connection can die silently: no error, just nothing. So one side sends a small ping every N seconds (say 30), and the other answers. If a connection misses, say, 3 in a row, treat it as dead: close it and clean up. Heartbeats also stop proxies and load balancers from closing "idle" connections. (WebSockets have built-in ping and pong frames; with SSE, send a comment line like : ping.) At 1 million connections and a 30-second interval, that's about 33,000 tiny pings a second, so keep them cheap.

Reconnects with backoff and jitter. When a server restarts, every client it held reconnects. If all 100,000 retry after exactly 1 second, the fresh server gets 100,000 connections in one second and may fall over again: a thundering herd. Instead:

• Exponential backoff: wait 1 s, then 2, 4, 8 ... up to a cap like 60 s.
• Random jitter: pick a random wait within that range. In the chapter's code, spreading 100,000 reconnects over 30 seconds brings the peak down to about 3,400 a second.

Figure: spread reconnects out, then resume from the last event the client saw.

Catching up on missed messages. While disconnected, the client misses events. Make every event carry an increasing ID (or sequence number) per user or per order, and on reconnect, the client says "the last one I saw was 2". The server sends everything after it, from a short buffer or the database. SSE does this for you with the Last-Event-ID header; with WebSockets, you build it. The client ignores events it has already seen, so a duplicate is harmless.

In the chapter's code, a phone drops after "packed". The buggy client reconnects and only hears "delivered", missing two updates; the fixed client asks for everything after event 2 and ends up with all five, in order, without duplicates.

And when in doubt, re-fetch. After a long disconnection, simply reloading the current state ("GET /orders/881") is often simpler than replaying hundreds of events.

Memory code: "Ping to check, wait a random bit to redial, then ask 'where were we?'"

---