Art of Codeartofcode
Lesson 7 of 16

6. Vertical vs Horizontal Scaling

1
The picture to keep in your head
2
6.1 What does "scaling" actually mean?
3
6.2 What is vertical scaling?
4
6.3 What is horizontal scaling?
5
6.4 Why does scaling out need stateless servers?
6
6.5 If servers are stateless, where does the state go?
7
6.6 What is autoscaling, and how do you get it right?
8
6.7 What about the database? Why is it the hardest part to scale?
9
6.8 What does each kind of scaling really cost?
10
6.9 So when do you choose which?
11
Memory card and where the analogy breaks
12
Practice: five levels
13
Interview questions
14
Solutions
15
Sources
16
Code lab
6. Vertical vs Horizontal Scaling

6.6 What is autoscaling, and how do you get it right?

3 min read+10 XP

In the kitchen: You don't pay ten chefs all day for a two-hour dinner rush. You bring in extra chefs for the rush and send them home after. Two things go wrong if you're careless: a new chef needs time to get ready before cooking, and if you hire and fire every ten minutes, the kitchen is chaos.

In the system: Autoscaling automatically adds servers when load rises and removes them when it falls, between a minimum and a maximum.

Ways to decide when:

  • Target tracking: "keep average CPU around 60%" or "keep about 300 requests a second per server". The autoscaler adds or removes servers to stay near the target.
  • Scheduled: "at 7:55 pm on sale day, have at least 67 servers". For events you know are coming.
  • Predictive: some platforms forecast from past days and scale ahead of the usual daily curve.

The rules that make it work:

  1. New servers aren't instant. Starting a machine, loading the app and warming its caches can take minutes. A reactive autoscaler is always behind a sudden spike.
  2. Scale out fast, scale in slowly. Leave a gap between the "add" and "remove" thresholds (for example, add above 70%, remove below 40%) and a cooldown after each change. Otherwise the fleet flaps: add a server, load drops, remove it, load rises, add it again (the Level 4 bug).
  3. Pick a metric that tracks the real limit. CPU works for compute-heavy services. For a service that mostly waits on the database, request count or latency is better. For workers, use queue length (Chapter 22).
  4. Set a sensible maximum, so a bug or an attack doesn't create a thousand servers, and so you don't overwhelm the database behind them.

a 10x spike: reactive scaling drops about 3 million requests during warm-up; scaling ahead drops none.

Figure: the chapter's simulation. With a 3-minute warm-up, reacting to the Book Fest spike loses about 3 million requests. Scaling ahead loses none.

For the Book Fest: the start time is known, so schedule the capacity before it, and keep target tracking on as a safety net for surprises.

Memory code: "Scale out fast, scale in slow, and schedule what you know."