Art of Codeartofcode
Lesson 13 of 17

7. Load Balancers, Reverse Proxies and API Gateways

1
The picture to keep in your head
2
7.1 Why do we need a load balancer?
3
7.2 What's the difference between a layer 4 and a layer 7 load balancer?
4
7.3 How does the load balancer choose a server?
5
7.4 How does the load balancer know a server is dead?
6
7.5 What are sticky sessions, and should you use them?
7
7.6 What is SSL termination?
8
7.7 If everything goes through the load balancer, what if it fails?
9
7.8 What's the difference between a forward proxy and a reverse proxy?
10
7.9 What is an API gateway?
11
7.10 How do you balance traffic across regions?
12
Memory card and where the analogy breaks
13
Practice: five levels
14
Interview questions
15
Solutions
16
Sources
17
Code lab
7. Load Balancers, Reverse Proxies and API Gateways

Practice: five levels

3 min read+10 XP

Assignments

Five levels. Do them in order. Answers are in "Solutions" at the end, so try first.

Level 1: Think (no code)

A. L4 or L7? Explain in one line.

  1. Spread connections across 6 PostgreSQL read replicas.
  2. Send /api/* to API servers and everything else to the web servers.
  3. A multiplayer game server using UDP.
  4. Make sessions sticky using a cookie.

B. Which algorithm?

  1. 10 identical servers, every request takes about 20 ms.
  2. Half the servers are new machines with twice the CPU.
  3. A search API where some queries take 5 ms and others take 5 seconds.
  4. Each server keeps a large per-user cache in memory that's slow to rebuild.

Level 2: Calculate

  1. Health checks run every 10 s with an unhealthy threshold of 3. In the worst case, about how long does traffic keep going to a crashed server? The site takes 3,000 requests a second across 10 servers. Roughly how many requests fail before the server is removed (no retries)?
  2. You change the checks to every 5 s with a threshold of 2. Redo question 1.
  3. Two servers have weight 2 and two have weight 1. They share 6,000 requests a second. How many does each get?
  4. During the sale, the load balancer handles 20,000 requests a second, each taking 100 ms end to end. Using Little's law (Chapter 5), how many requests are in progress through the load balancer at any moment?

Level 3: Build

In Python, write:

  1. RoundRobin, WeightedRoundRobin (spreading picks smoothly), LeastConnections and IPHash classes, each with a pick() method.
  2. A HealthChecker that marks a server down after 3 failures in a row and up after 2 passes in a row.
  3. A simulation where 90% of requests take 2 time units and 10% take 60, comparing the busiest server's in-progress count under round robin and least connections.

Check that:

  • Round robin cycles a, b, c, a, b, c.
  • Weighted round robin with weights 2 and 1 gives a 200/100 split over 300 picks.
  • The same IP always goes to the same server.
  • The health checker flips at exactly the right moments.
  • Least connections keeps the busiest server lower than round robin.

Level 4: Debug

The bookstore keeps a big per-user recommendations cache in each app server's memory and uses IP hash so users stick to their server:

python

After the Book Fest, the team scales in from 4 servers to 3. Suddenly every server is slow for an hour and the database is overloaded.

  1. Find the bug. Roughly what fraction of users moved to a different server, and why does that overload the database?
  2. Fix it. Give two different fixes.

Level 5: Design

Design the front door for the bookstore during the Book Fest: 20,000 requests a second at peak, a website and a mobile app, 68 app servers split across 3 availability zones, separate catalogue, cart and payments services.

Deliver:

  1. The path of a request from the user to a service, naming each component.
  2. Where TLS is terminated, and whether traffic is encrypted inside.
  3. The balancing algorithm and health-check settings, with reasons.
  4. How you avoid a single point of failure at the front door.
  5. What the API gateway does, and one thing it must not do.