36. Design a Web Crawler
1
The interview prompt2
36.1 What do you ask first?3
36.2 How big is this, really?4
36.3 What are the doors?5
36.4 What do we store?6
36.5 What does the whole system look like?7
36.6 How does the URL frontier work?8
36.7 How do politeness and robots.txt work?9
36.8 How do you de-duplicate with Bloom filters?10
36.9 How do you crawl with many machines?11
36.10 What would you say in the last five minutes?12
Score yourself13
Variations to practise14
Sources15
Code lab36. Design a Web Crawler
36.3 What are the doors?
3 min read+10 XP
Step 3: API and data model (10 minutes)
A crawler is mostly internal, so its "API" is the interfaces between its parts:
frontier.add(url, priority) # from seeds, extracted links, re-crawl scheduler
frontier.next() -> url # only when that host's politeness timer allows
fetcher.fetch(url) -> status, headers, body (bounded size and time)
parser.links(body, base_url) -> [url]
store.put(url, fetched_at, content_hash, compressed_body)
Plus admin endpoints: add seeds, pause or slow down a host, block a domain, see crawl stats.