Art of Codeartofcode
Lesson 6 of 15

32. Design a Rate Limiter as a Service

1
The interview prompt
2
32.1 What do you ask first?
3
32.2 How big is this, really?
4
32.3 What are the doors?
5
32.4 What do we store?
6
32.5 What does the whole system look like?
7
32.6 How do you pick the algorithm?
8
32.7 How do counters work in Redis?
9
32.8 What race conditions do you have to handle?
10
32.9 How do limits work across regions?
11
32.10 What would you say in the last five minutes?
12
Score yourself
13
Variations to practise
14
Sources
15
Code lab
32. Design a Rate Limiter as a Service

32.5 What does the whole system look like?

3 min read+10 XP

Step 4: High-level design (10 minutes)

clients reach a gateway; its limiter client checks counters in a Redis cluster with one script; rules come from a rules service; metrics flow out asynchronously.

Figure: the final design. The green path is on every request, so it has exactly one network hop to Redis.

A check, step by step:

  1. A request reaches the API gateway (or a service). Its embedded limiter client finds the matching rule in its local rule cache.
  2. The client runs one Lua script on the Redis shard that owns the key: refill, check, spend, return remaining and retry-after (32.7).
  3. The gateway either continues or replies 429 with Retry-After and RateLimit-* headers.
  4. The client emits counters ("allowed", "limited" per rule) asynchronously to metrics.

A rule change, step by step:

  1. A team edits its rule through the admin API or config-as-code; it's validated and stored with a new version.
  2. Limiter clients pick up the change within seconds (poll or push) and switch atomically to the new version.

Interviewer pushback:

  • "Walk me through Redis being slow or down." → The client's call has a 5 ms timeout and a circuit breaker (Chapter 29). On timeout: rules marked open allow the request and apply a coarse local token bucket (limit ÷ number of instances, with slack) as a backstop; rules marked closed (login, OTP) deny or use a strict local limit. Either way, the limiter never makes a request wait on a dead dependency.
  • "Library, sidecar, or a separate service?" → Library is fastest (no extra hop) but must exist per language. A sidecar (or the gateway itself) gives one implementation for every language at the cost of a local hop. A separate network service adds a full hop to every request, so I'd offer it only as a fallback API.
  • "How do you stop one team's rules from hurting others?" → Per-team key prefixes, limits on rule count and key cardinality per team, and if needed separate Redis clusters for the heaviest tenants (a bulkhead, Chapter 29).

Memory code: "Rules in memory, counters in Redis, one hop per check, never wait on a dead limiter."

The interviewer usually picks one or two. A strong candidate offers: "The interesting parts are which algorithm per rule, keeping counters correct under concurrency, and limits across regions. Which would you like?"