Art of Codeartofcode
Lesson 4 of 12

21. Polling vs WebSockets vs Push

1
The picture to keep in your head
2
21.1 What's the problem: the server has news for you?
3
21.2 What are the options?
4
21.3 How do you scale millions of open connections?
5
21.4 How do heartbeats, reconnects and missed messages work?
6
21.5 How do you choose?
7
Memory card and where the analogy breaks
8
Practice: five levels
9
Interview questions
10
Solutions
11
Sources
12
Code lab
21. Polling vs WebSockets vs Push

21.3 How do you scale millions of open connections?

3 min read+10 XP

At the gate: A call centre with thousands of open lines. Each operator can hold many quiet lines at once. When the kitchen has news for customer 881, someone has to know which operator is holding 881's line.

In the system: SSE and WebSockets mean one open connection per user, for minutes or hours. At 1 million users, that's 1 million connections held at once. Three problems:

1. Connections per server. Each idle connection costs memory (buffers, TLS state, your own per-user data) and a file descriptor. If one costs about 20 KB (an assumption: measure your own), 1 million connections need about 20 GB of memory across your servers. Event-driven servers (Node.js, Go, Netty, Erlang/Elixir) handle large numbers of mostly idle connections well. Raise OS limits like open file descriptors, and keep per-connection work tiny.

2. Which server holds which user? The order service knows "order 881 is packed", but Asha's connection is on gateway 1 of 10. Two common patterns:

  • Pub/sub backplane: every gateway subscribes to a shared channel (Redis pub/sub, Kafka, NATS); the order service publishes "user Asha: packed", and the gateway holding Asha delivers it (Chapter 23).
  • Connection registry: store "Asha → gateway 1" in a fast store and send the message to that gateway directly. More efficient at huge scale, but the registry must stay accurate as users reconnect.

phones connect to gateways; the order service publishes to a backplane, which routes to the right gateway.

Figure: gateways only hold sockets; a backplane routes each message to whoever holds the user.

3. Load balancing and deploys. A long-lived connection sticks to one server, so:

  • Load balancers must support long timeouts and the WebSocket upgrade; balance by connection count, not requests.
  • A new server gets no traffic until clients reconnect; a deploy drops thousands of connections at once. Drain gracefully: stop accepting new connections, tell clients to reconnect elsewhere, then shut down.
  • Separate the connection gateways from business logic, so you can deploy business code without dropping everyone's connection.

Memory code: "Gateways hold the lines; the backplane knows who's on which."