Art of Codeartofcode
Lesson 13 of 15

36. Design a Web Crawler

1
The interview prompt
2
36.1 What do you ask first?
3
36.2 How big is this, really?
4
36.3 What are the doors?
5
36.4 What do we store?
6
36.5 What does the whole system look like?
7
36.6 How does the URL frontier work?
8
36.7 How do politeness and robots.txt work?
9
36.8 How do you de-duplicate with Bloom filters?
10
36.9 How do you crawl with many machines?
11
36.10 What would you say in the last five minutes?
12
Score yourself
13
Variations to practise
14
Sources
15
Code lab
36. Design a Web Crawler

Variations to practise

3 min read+10 XP

Chapters used

Approach (1), Estimation (2), DNS (4), Blob storage (11), Search and indexing (19), Bloom filters (20), Message queues (22), Rate limiting and 429s (25), Consistent hashing (28), Failures and circuit breakers (29).

Try these variations

Each is a common follow-up. Answer in two minutes, then check the sketch.

  1. "Crawl only book publishers' sites for prices, every hour." → A focused crawler: a fixed list of hosts, their sitemaps, conditional requests to skip unchanged pages, and a price parser per site.
  2. "Render JavaScript-heavy pages." → A separate pool of headless browsers, used only for pages that need it (detected by empty HTML), with much lower throughput and its own queue.
  3. "Detect near-duplicate pages." → SimHash fingerprints per page; pages within a few bits of each other are near-duplicates; keep one canonical copy (also honour rel=canonical).
  4. "10 billion pages a month." → 10× the fleet and bandwidth, Bloom filter and url_state sharded further, more aggressive prioritisation so the most valuable pages come first.