36.5 What does the whole system look like?

Step 4: High-level design (10 minutes)

Figure: the final design is one loop, frontier → fetch → parse → filter → frontier, run on many nodes.

The crawl loop, step by step:

1. The frontier hands out the next URL whose host is allowed to be fetched now (36.6).
2. A fetcher checks the host's cached robots.txt (36.7), resolves the host through a DNS cache, and downloads the page with a timeout and a size limit.
3. The page is parsed; its normalised text is hashed, and if the content was already seen, it's recorded as a duplicate and not stored again (36.8).
4. New pages go to object storage, and an event tells the indexer (Chapter 19).
5. The link extractor pulls out links, normalises them, drops ones that are filtered (wrong type, blocked domain, too deep, too long), and checks the seen-URL filter. New URLs go into the frontier with a priority.
6. A re-crawl scheduler puts known URLs back into the frontier when they're due, more often for pages that change often.

Interviewer pushback:

• "What about spider traps, like a calendar with infinite 'next month' links?" → Limit URL length and path depth per site, cap pages per host per crawl, detect repeating path patterns, and lower priority for URLs that keep producing near-identical pages.
• "A site is slow or returning errors." → Back off that host: longer delays, fewer concurrent connections, and pause it after repeated failures (Chapter 29's circuit breaker, per host).

Memory code: "Frontier, fetch, parse, filter, back to the frontier."

---

Offer: "The interesting parts are the frontier, politeness and robots.txt, de-duplication, and splitting the crawl across machines. Which would you like?"