7. Load Balancers, Reverse Proxies and API Gateways
Solutions
3 min read+10 XP
Solutions
Level 1
reveal
Level 2
reveal
Level 3
reveal
Level 4
reveal
Level 5 (sketch)
reveal
No tracks available
Lofi Player
1Show the Level 1 solution
2**A.**
31. **L4.** It's the PostgreSQL protocol, not HTTP; just spread TCP connections.
42. **L7.** Routing by path needs to read the HTTP request.
53. **L4.** UDP, and there's no HTTP to read.
64. **L7.** Setting and reading a cookie needs HTTP.
78**B.**
91. **Round robin.** Identical servers, similar requests.
102. **Weighted round robin**, weight 2 for the new machines.
113. **Least connections.** Request lengths vary hugely.
124. **IP hash, or better, consistent hashing on the user ID** (Chapter 28), so users keep landing where their cache is, and adding or removing a server moves few of them.
Show the Level 2 solution
1. Worst case about **30 seconds** (a crash just after a passed check, then three failed checks 10 s apart). Each server gets 3,000 ÷ 10 = 300 requests a second, so about 300 × 30 = **9,000 failed requests**.
2. About **10 seconds** (2 checks × 5 s) → 300 × 10 = **3,000 failed requests**. Faster detection, at the risk of flagging a server that was briefly slow.
3. Total weight = 2 + 2 + 1 + 1 = 6. Each weight-2 server gets 6,000 × 2 ÷ 6 = **2,000 a second**; each weight-1 server gets **1,000**.
4. Little's law: 20,000 × 0.1 = **2,000 requests in progress**. The load balancer must handle at least that many open connections comfortably.
1Show the Level 3 solution
2Tested solution (Python 3), the key parts:
34```python
5class RoundRobin:
6 def __init__(self, servers):
7 self.servers, self.i = list(servers), 0
89 def pick(self, **_):
10 s = self.servers[self.i % len(self.servers)]
11 self.i += 1
12 return s
131415class WeightedRoundRobin:
16 """Smooth weighted round robin: spreads picks out instead of bursts."""
17 def __init__(self, weights):
18 self.weights = dict(weights)
19 self.current = {s: 0 for s in weights}
2021 def pick(self, **_):
22 total = sum(self.weights.values())
23 for s, w in self.weights.items():
24 self.current[s] += w
25 best = max(self.current, key=self.current.get)
26 self.current[best] -= total
27 return best
282930class LeastConnections:
31 def __init__(self, servers):
32 self.active = {s: 0 for s in servers} # updated as requests start and finish
3334 def pick(self, **_):
35 return min(self.active, key=self.active.get)
363738class HealthChecker:
39 def __init__(self, servers, unhealthy=3, healthy=2):
40 self.up = {s: True for s in servers}
41 self.fails = {s: 0 for s in servers}
42 self.passes = {s: 0 for s in servers}
43 self.unhealthy, self.healthy = unhealthy, healthy
4445 def record(self, server, ok):
46 if ok:
47 self.passes[server] += 1
48 self.fails[server] = 0
49 if not self.up[server] and self.passes[server] >= self.healthy:
50 self.up[server] = True
51 else:
52 self.fails[server] += 1
53 self.passes[server] = 0
54 if self.up[server] and self.fails[server] >= self.unhealthy:
55 self.up[server] = False
56```
5758All checks pass. In the simulation (4 servers, 20,000 requests), the busiest server peaks at **8** requests in progress with round robin and **5** with least connections. The full file, including `IPHash` and the simulation, is in `code/ch07_load_balancer.py`.
1Show the Level 4 solution
2**The bug:** `hash % len(servers)` depends on the **number of servers**. When it changes from 4 to 3, almost every user's result changes.
341. In the tested simulation, about **75%** of users moved to a different server. Each of them now lands on a server with an **empty** recommendations cache for them, so every one of those users' requests misses the cache and goes to the database at the same time: a self-inflicted cache stampede (Chapter 13).
52. **Fixes:**
6 - **Consistent hashing** (Chapter 28): servers sit on a hash ring, and removing one only moves the users that were on it. In the simulation, about **24%** moved, close to the ideal 1 in 4.
7 - **Make the cache shared:** move the recommendations cache to Redis, so it doesn't matter which app server a user reaches (Chapter 6, section 6.5). Then plain round robin or least connections works.
8 - Either way, scale in **gradually**, one server at a time, while the database has headroom.
1Show the Level 5 (sketch) solution
21. **Path:** user → DNS (`shop.example.com`) → a managed cloud load balancer spread across 3 zones (L7) → API gateway → the right service (catalogue, cart, payments), each with its own pool of app servers across the 3 zones, behind internal load balancing.
32. **TLS:** terminate at the edge load balancer with managed certificates. **Re-encrypt** to the gateway and services, because payments data travels inside (and many compliance rules require it).
43. **Algorithm:** **least connections** (or least outstanding requests), since search and checkout requests vary in length. **Health checks:** every 5 seconds, unhealthy after 2 failures, healthy after 2 passes, on a lightweight readiness endpoint; **connection draining** of 30 seconds for deploys and scale-in.
54. **No single point of failure:** managed load balancer across zones (or an active-passive pair with a virtual IP if self-hosted), at least 2 gateway instances per zone, servers in all 3 zones, each zone able to absorb another's traffic.
65. **Gateway does:** token checks, rate limiting (especially on checkout and login), routing by path, request IDs, logging and metrics. **Must not:** contain business rules, like discount calculations or stock checks. Those belong in the services.