Art of Codeartofcode
Lesson 5 of 15

36. Design a Web Crawler

1
The interview prompt
2
36.1 What do you ask first?
3
36.2 How big is this, really?
4
36.3 What are the doors?
5
36.4 What do we store?
6
36.5 What does the whole system look like?
7
36.6 How does the URL frontier work?
8
36.7 How do politeness and robots.txt work?
9
36.8 How do you de-duplicate with Bloom filters?
10
36.9 How do you crawl with many machines?
11
36.10 What would you say in the last five minutes?
12
Score yourself
13
Variations to practise
14
Sources
15
Code lab
36. Design a Web Crawler

36.4 What do we store?

3 min read+10 XP
pages (object storage, Chapter 11) key: hash(url) + fetch time → compressed HTML url_state (key-value store, sharded by host) url (normalised), last_fetched, last_status, content_hash, next_due, priority, depth host_state host → robots.txt (cached ≤ 24 h), crawl delay, error rate, DNS answer + TTL seen_urls Bloom filter (per crawler node, for its hosts) content_seen content_hash → first URL (exact-duplicate detection)

Interviewer pushback:

  • "Why keep url_state if you have the Bloom filter?" → The Bloom filter only answers "probably seen?" cheaply. url_state holds what re-crawling needs: when we fetched it, whether it changed, and when it's due again.

Memory code: "Pages in object storage, state per URL and per host, a filter for 'seen?'."