36. Design a Web Crawler
1
The interview prompt2
36.1 What do you ask first?3
36.2 How big is this, really?4
36.3 What are the doors?5
36.4 What do we store?6
36.5 What does the whole system look like?7
36.6 How does the URL frontier work?8
36.7 How do politeness and robots.txt work?9
36.8 How do you de-duplicate with Bloom filters?10
36.9 How do you crawl with many machines?11
36.10 What would you say in the last five minutes?12
Score yourself13
Variations to practise14
Sources15
Code lab36. Design a Web Crawler
36.1 What do you ask first?
3 min read+10 XP
Step 1: Requirements (5 minutes)
| Letter | Candidate asks | Interviewer answers |
|---|---|---|
| Scope | "HTML only, or images and PDFs too? Do we re-crawl pages that change? Do we render JavaScript? Do we respect robots.txt?" | "HTML only for now. Yes, re-crawl, more often for pages that change. No JavaScript rendering. Yes, robots.txt." |
| Count | "How many pages, how fast?" | "About a billion pages a month." |
| Access pattern | "Is the web we crawl a few big sites or many small ones?" | "Millions of sites, from huge to tiny." |
| Latency | "How fresh must the index be?" | "News sites within hours; most pages within weeks." |
| Exactness | "Is missing some pages OK? Duplicates?" | "Missing a few is fine. Don't waste effort on duplicates." |
Agreed requirements:
- Functional (verbs): start from seed URLs; fetch pages; extract links; decide what to fetch next; store pages for the indexer; re-crawl by how often pages change.
- Non-functional (adjectives): polite (respect robots.txt, never overload a site), scalable to a billion pages a month on many machines, resilient to bad sites (slow servers, traps, huge pages, malformed HTML), efficient (no duplicate URLs or content), prioritised (important and fresh pages first).
Interviewer pushback:
- "Why is politeness a requirement and not a nice-to-have?" → A crawler fetching hundreds of pages a second can knock a small site over. Being impolite gets you blocked, harms site owners, and breaks the rules (robots.txt) the web runs on.
Memory code: "Ask how many pages, how fresh, and how gently."