32. Design a Rate Limiter as a Service
1
The interview prompt2
32.1 What do you ask first?3
32.2 How big is this, really?4
32.3 What are the doors?5
32.4 What do we store?6
32.5 What does the whole system look like?7
32.6 How do you pick the algorithm?8
32.7 How do counters work in Redis?9
32.8 What race conditions do you have to handle?10
32.9 How do limits work across regions?11
32.10 What would you say in the last five minutes?12
Score yourself13
Variations to practise14
Sources15
Code lab32. Design a Rate Limiter as a Service
32.1 What do you ask first?
3 min read+10 XP
Step 1: Requirements (5 minutes)
The candidate runs through SCALE (Chapter 1, section 1.5) and writes the answers down.
| Letter | Candidate asks | Interviewer answers |
|---|---|---|
| Scope | "Who calls it: our gateway, our services, or both? What do teams limit by: user, API key, IP? Do we need different algorithms? Quotas per day too?" | "Both the API gateway and internal services. Limits by any key the team chooses. Bursty API traffic and some daily quotas." |
| Count | "How many checks a second at peak? How many distinct keys?" | "About 200,000 checks a second at peak. Maybe 10 million active keys." |
| Access pattern | "Is every request checked, or only some?" | "Every request to a limited endpoint." |
| Latency | "How much latency can the check add? What if the limiter is down?" | "A few milliseconds at most. If it's down, the product must keep working." |
| Exactness | "Must limits be exact, or is a small overshoot fine? Across regions too?" | "Small overshoot is fine for API limits. Login and OTP limits should be tight. We run in three regions." |
Agreed requirements:
- Functional (verbs): teams define rules (key, limit, window, burst); callers check a request and get allowed or not, with remaining and retry-after; teams see what's being limited.
- Non-functional (adjectives): fast (under ~5 ms added, p99), highly available and fail-safe (a limiter outage must not become a product outage), roughly accurate (tight for security rules), multi-tenant (one team's rules can't hurt another), multi-region.
Interviewer pushback:
- "Why not let each team keep doing it themselves?" → They each reinvent the same races and failure handling (Chapter 25), with inconsistent headers and no shared visibility. A shared service fixes it once.
- "What does 'fail-safe' mean here?" → For most rules, fail open: if we can't check, allow the request and rely on a coarse local backstop. For security rules (login, OTP), fail closed or fall back to a strict local limit. Each rule declares which.
Memory code: "Ask who calls, how fast, and what happens when it's down."