Art of Codeartofcode
Lesson 11 of 15

36. Design a Web Crawler

1
The interview prompt
2
36.1 What do you ask first?
3
36.2 How big is this, really?
4
36.3 What are the doors?
5
36.4 What do we store?
6
36.5 What does the whole system look like?
7
36.6 How does the URL frontier work?
8
36.7 How do politeness and robots.txt work?
9
36.8 How do you de-duplicate with Bloom filters?
10
36.9 How do you crawl with many machines?
11
36.10 What would you say in the last five minutes?
12
Score yourself
13
Variations to practise
14
Sources
15
Code lab
36. Design a Web Crawler

36.10 What would you say in the last five minutes?

3 min read+10 XP

Step 6: Wrap-up (5 minutes)

  • Bottlenecks: network concurrency (async fetchers), DNS (local caching resolvers), and politeness limits on big sites (more parallelism only where sites allow it).
  • Single points of failure: none central: nodes own hosts on a ring; url_state and storage are replicated.
  • Resilience: timeouts, size limits, trap detection, per-host circuit breakers, and never trusting HTML to be well formed.
  • Freshness: re-crawl intervals learned per page (shorter if the content hash keeps changing), sitemaps and RSS feeds as hints, conditional requests (If-Modified-Since, ETags) to skip unchanged pages cheaply.
  • Next with more time: JavaScript rendering for pages that need it (a separate, slower rendering pool), images and PDFs, language detection, and spam detection.

Memory code: "Gentle to every site, fast across all of them."