Art of Codeartofcode
Lesson 14 of 15

36. Design a Web Crawler

1
The interview prompt
2
36.1 What do you ask first?
3
36.2 How big is this, really?
4
36.3 What are the doors?
5
36.4 What do we store?
6
36.5 What does the whole system look like?
7
36.6 How does the URL frontier work?
8
36.7 How do politeness and robots.txt work?
9
36.8 How do you de-duplicate with Bloom filters?
10
36.9 How do you crawl with many machines?
11
36.10 What would you say in the last five minutes?
12
Score yourself
13
Variations to practise
14
Sources
15
Code lab
36. Design a Web Crawler

Sources

3 min read+10 XP

Source notes

  • Robots Exclusion Protocol: IETF RFC 9309, September 2022 (user-agent groups, longest-match rule with Allow winning ties, caching, handling of unavailable and unreachable robots.txt). Crawl-delay is not part of RFC 9309 and Google Search does not support it: Google Search Central blog, 2025. Checked 2026-10-08.
  • Python's urllib.robotparser evaluates rules in order of appearance: demonstrated in the chapter's code.
  • Front and back queues for prioritised, polite crawling: Allan Heydon and Marc Najork, "Mercator: A Scalable, Extensible Web Crawler", World Wide Web, 1999.
  • Near-duplicate detection with SimHash: Moses Charikar, "Similarity Estimation Techniques from Rounding Algorithms", STOC 2002; Gurmeet Singh Manku et al., "Detecting Near-Duplicates for Web Crawling", WWW 2007.
  • Page sizes, compression ratio, fetch time and URL counts are interview assumptions agreed in Step 1, not measurements of the web. Arithmetic computed 2026-10-08.
  • Frontier politeness, priority, robots.txt behaviour, URL normalisation, Bloom filter and host partitioning: tested 2026-10-08 (code/ch36_web_crawler.py, all checks pass).