Art of Codeartofcode
Lesson 1 of 15

36. Design a Web Crawler

1
The interview prompt
2
36.1 What do you ask first?
3
36.2 How big is this, really?
4
36.3 What are the doors?
5
36.4 What do we store?
6
36.5 What does the whole system look like?
7
36.6 How does the URL frontier work?
8
36.7 How do politeness and robots.txt work?
9
36.8 How do you de-duplicate with Bloom filters?
10
36.9 How do you crawl with many machines?
11
36.10 What would you say in the last five minutes?
12
Score yourself
13
Variations to practise
14
Sources
15
Code lab
36. Design a Web Crawler

The interview prompt

3 min read+10 XP

This chapter is a full interview round, played out step by step. The interviewer's lines are in quotes. At each step you'll see what a strong candidate says, the pushback the interviewer throws, and how to answer it. At the end, grade yourself with the rubric.

How to use this chapter

  1. Read only the prompt below. Then stop.
  2. Set a 45-minute timer and do the whole round yourself, on paper or a whiteboard, out loud. Use the framework from Chapter 1: READ, then Draw, Dive, Defend.
  3. Then read the walkthrough and compare. Score yourself with the rubric at the end.

A crawler looks simple: download a page, find its links, download those. The interview is about everything that goes wrong at scale: being polite to the sites you visit, never fetching the same thing twice, and splitting the work across many machines.

The prompt

"Design a web crawler that collects about a billion pages a month, to feed a search engine."

Stop here. Start your timer.