Score yourself

Give yourself 0, 1 or 2 points per row.

Area	0 points	1 point	2 points
Requirements	Started drawing	Pages and links	Politeness, freshness, duplicates and resilience agreed
Estimates	None	Numbers, not used later	Numbers that drove async fetching, object storage and a Bloom filter
API and data	Skipped	Some components	Frontier, fetcher and store interfaces; url_state and host_state
High-level design	Loop without details	Loop with boxes	Loop walked, with DNS, robots.txt, dedupe and re-crawl
Deep dive	Hand-waved	One topic	Frontier with per-host politeness, RFC robots rules, dedupe or host-based partitioning
Trade-offs	None named	Named when asked	Raised unprompted (Bloom false positives, split by host, trap limits)
Communication	Long silences	Mostly clear	Thought out loud, checked in, managed time

12 to 14: strong hire signal. 8 to 11: solid, polish the weak rows. Under 8: redo the round in a week, timer on.

The bar: fresher vs SDE-2

	Fresher	SDE-2
Frontier	A queue of URLs (BFS)	Priority front queues, per-host back queues, next-allowed heap
Politeness	"Respect robots.txt"	RFC 9309 matching, caching rules, adaptive delays, back-off
Dedupe	A set of visited URLs	Normalisation, Bloom filter sizing, content hashes, near-duplicates
Scale	Many workers	Partition by host on a consistent-hash ring, forwarded links
Resilience	Not mentioned	Timeouts, size limits, traps, per-host breakers, re-crawl scheduling

Common mistakes

A single BFS queue, so hundreds of requests hit one site at once.
Ignoring robots.txt, or parsing it with first-match rules.
No URL normalisation, so the same page is fetched under dozens of URLs.
Splitting work by URL, breaking per-host politeness.
Storing pages in a database instead of object storage.

Quick check

Five questions on this chapter. Pick an answer, then check it.

quiz

1[
2 {
3  "q": "What decides WHEN a crawler can fetch the next page from a site?",
4  "options": [
5   "The global priority only",
6   "The host's politeness clock (crawl delay), separate from page priority",
7   "Random",
8   "The page size"
9  ],
10  "answer": 1,
11  "explain": "Priority decides what; the host's clock decides when."
12 },
13 {
14  "q": "How do you avoid fetching the same URL twice cheaply?",
15  "options": [
16   "Store every URL in a list and scan it",
17   "Normalise the URL, check a Bloom filter, and hash content to catch duplicates",
18   "Never revisit any site",
19   "Ask the site"
20  ],
21  "answer": 1,
22  "explain": "Normalise the address, ask the Bloom filter, hash the content."
23 },
24 {
25  "q": "With many crawler machines, how should work be split?",
26  "options": [
27   "By page, randomly",
28   "By site (host), so one owner keeps each site's pace gentle",
29   "By file type",
30   "By time"
31  ],
32  "answer": 1,
33  "explain": "Split by site, not by page: one owner, one gentle knock at a time."
34 },
35 {
36  "q": "What should a crawler do with robots.txt?",
37  "options": [
38   "Ignore it",
39   "Fetch and respect it, back off on errors, and stop when asked",
40   "Read it once a year",
41   "Only for big sites"
42  ],
43  "answer": 1,
44  "explain": "Read the sign on the door, knock gently, and leave when asked."
45 },
46 {
47  "q": "Where do fetched pages go?",
48  "options": [
49   "The URL database",
50   "Object storage, with per-URL state in a database",
51   "Memory only",
52   "The frontier"
53  ],
54  "answer": 1,
55  "explain": "Pages in object storage, state per URL and per host, a filter for 'seen?'."
56 }
57]

36. Design a Web Crawler

Score yourself

Score yourself

The bar: fresher vs SDE-2

Common mistakes

Quick check

Platform

Company

Legal