Art of Codeartofcode
Lesson 12 of 16

6. Vertical vs Horizontal Scaling

1
The picture to keep in your head
2
6.1 What does "scaling" actually mean?
3
6.2 What is vertical scaling?
4
6.3 What is horizontal scaling?
5
6.4 Why does scaling out need stateless servers?
6
6.5 If servers are stateless, where does the state go?
7
6.6 What is autoscaling, and how do you get it right?
8
6.7 What about the database? Why is it the hardest part to scale?
9
6.8 What does each kind of scaling really cost?
10
6.9 So when do you choose which?
11
Memory card and where the analogy breaks
12
Practice: five levels
13
Interview questions
14
Solutions
15
Sources
16
Code lab
6. Vertical vs Horizontal Scaling

Practice: five levels

3 min read+10 XP

Assignments

Five levels. Do them in order. Answers are in "Solutions" at the end, so try first.

Level 1: Think (no code)

A. Scale up or scale out first? One line each.

  1. A small company's Postgres database is at 85% CPU on 4 cores.
  2. A stateless API is at 80% CPU on 10 servers and traffic grows 20% a month.
  3. A video-resizing service where each job is independent.
  4. A single-server app that keeps logged-in sessions in memory, before a big launch next week.

B. "If I kill this server right now, does any user lose anything?" Answer for each:

  1. An app server with sessions in Redis and uploads sent straight to object storage.
  2. An app server that saves uploaded profile photos to its own disk.
  3. A worker that has taken a job off a queue but hasn't finished it (and hasn't acknowledged it).

Level 2: Calculate

Book Fest assumptions: normal peak 2,000 requests a second, sale peak 20,000. One server handles 500 a second; run at 60%; keep 1 spare.

  1. How many servers for a normal evening? For the sale?
  2. 10% of every request's time is spent waiting on one shared lock. With 10 servers, what's the best possible speed-up? With 100? What's the limit?
  3. The sale peak lasts 2 hours a day; the rest of the day needs the normal fleet. Compare server-hours a day for "always run the sale fleet" vs autoscaling.
  4. New servers take 3 minutes to start. The sale starts with no warning at 20,000 a second while 7 servers (3,500 a second) are running. Roughly how many requests are dropped before new servers are ready?

Level 3: Build

Write a Python autoscaler simulation:

  • Input: requests a second for each minute; per-server capacity; target utilisation; minimum and maximum servers; a warm-up delay before new servers serve; optional scheduled minimum from a given minute on.
  • Each minute: decide how many servers you want, order new ones (ready after warm-up), remove extras only when nothing is starting, and count requests above capacity as dropped.

Check that:

  1. Steady 2,000 a second settles at 7 servers and drops nothing.
  2. A sudden jump to 20,000 drops requests while servers warm up, then settles at 67.
  3. Scheduling 67 servers 5 minutes before the jump drops nothing.

Level 4: Debug

The autoscaler keeps adding and removing a server every minute, even though traffic is steady. Deploy logs are a mess and users see occasional errors when servers are removed.

python

The steady load is worth about 2.6 servers of CPU (so 4 servers ≈ 65%, 5 servers ≈ 52%).

  1. Find the bug. Walk through the first four minutes starting at 4 servers.
  2. Fix it.

Level 5: Design

The bookstore today: one large server runs the app and the database, keeps sessions in memory, and stores uploaded book covers on its local disk. The Book Fest is in 3 weeks, at 10 times normal traffic.

Deliver:

  1. What you change so the app tier can scale out (list each kind of state and its new home).
  2. The new layout of the system.
  3. How you scale the database for the sale, step by step.
  4. Your autoscaling plan for sale day (scheduled and reactive), including min, max and the metric.
  5. Two risks on sale day and how you'd test for them beforehand.