36. Design a Web Crawler
1
The interview prompt2
36.1 What do you ask first?3
36.2 How big is this, really?4
36.3 What are the doors?5
36.4 What do we store?6
36.5 What does the whole system look like?7
36.6 How does the URL frontier work?8
36.7 How do politeness and robots.txt work?9
36.8 How do you de-duplicate with Bloom filters?10
36.9 How do you crawl with many machines?11
36.10 What would you say in the last five minutes?12
Score yourself13
Variations to practise14
Sources15
Code lab36. Design a Web Crawler
36.2 How big is this, really?
3 min read+10 XP
Step 2: Estimates (5 minutes)
Assume 100 KB per HTML page, 4× compression for storage, about 1 second per fetch (network, server time and transfer), and about 10 billion distinct URLs discovered over time.
| What | Working | Result |
|---|---|---|
| Pages a second | 1 billion ÷ (30 × 86,400) | ≈ 386 a second (plan for ~800 at peak) |
| Bandwidth | 386 × 100 KB | ≈ 309 Mbit/s average |
| Concurrent fetches | 386 × 1 s (Little's law) | ≈ 400 in flight; plan for thousands, so slow sites don't block |
| Raw pages a month | 1 billion × 100 KB | ≈ 100 TB |
| Stored (compressed) | 100 TB ÷ 4 | ≈ 25 TB a month |
| Seen-URL filter | 10 billion × 9.6 bits (1% false positives) | ≈ 12 GB as a Bloom filter |
What the numbers say, out loud:
- "Hundreds of pages a second is modest for machines, but thousands of concurrent connections means async I/O, not a thread per request."
- "Storage grows fast: tens of TB a month, so pages go to object storage (Chapter 11), not a database."
- "10 billion URLs don't fit in a hash set on one machine cheaply, but a Bloom filter of about 12 GB does (Chapter 20)."
Interviewer pushback:
- "Why not just use one fast machine?" → Bandwidth, concurrency and the URL sets would fit a few big machines, but one machine is a single point of failure and its IP gets rate-limited by sites. A modest fleet spreads both load and politeness.
Memory code: "Hundreds of pages a second, thousands in flight, terabytes a month."