36.9 How do you crawl with many machines?

The key decision is how to split the work. Split by host, not by URL:

Figure: one owner per site keeps politeness, robots.txt and DNS local.

• Owner = hash(host), placed on a consistent-hash ring (Chapter 28), so adding or removing a node moves only a small share of hosts.
• Each node keeps its hosts' back queues, politeness timers, robots.txt and DNS answers locally: no coordination needed to be polite.
• When a node extracts a link to another node's host, it forwards it (batched) to the owner, directly or through a shared queue (Chapter 22).
• The seen-URL filter is split the same way, so "seen?" is always a local check.

In the chapter's code, with 8 nodes, every URL of a given host lands on the same node. If you split by URL instead, five nodes could hit the same small site at once, and each would need its own copy of that site's robots.txt and timers.

Failure: if a node dies, its hosts move to the next node on the ring; their queues are rebuilt from url_state (due URLs) and the shared queue of forwarded links. A few pages may be fetched twice, which is harmless.

Memory code: "Split by site, not by page: one owner, one gentle knock at a time."

---