36.7 How do politeness and robots.txt work?

robots.txt is a file at the root of a site (/robots.txt) telling crawlers which paths they may fetch. It was standardised as RFC 9309 in 2022:

User-agent: * Disallow: /cart Disallow: /search Allow: /search/about User-agent: BadBot Disallow: /

Rules that matter:

• Use the group for your user agent, or * if none matches.
• The longest matching rule wins; if an Allow and a Disallow match equally, Allow wins. So /search/about is allowed above, and /search?q=dune isn't.
• Cache it, but don't use a cached copy for more than about a day; if robots.txt can't be fetched because the server is failing, assume you may not crawl; if it simply doesn't exist (a 4xx), you may crawl.

A real-world trap the chapter's code shows: Python's built-in urllib.robotparser applies rules in file order (first match wins), so it disallows /search/about in this example. A small RFC-style matcher (longest match wins) allows it. Test your parser against the standard, not just its examples.

Politeness beyond robots.txt:

• One connection per host (or a few for very large sites), with a delay between requests, adapted to how fast the site responds: if a page takes 2 seconds, wait longer.
• Crawl-delay is a common non-standard directive. It isn't part of RFC 9309; some crawlers honour it and Google doesn't. Honouring it is the polite choice.
• Identify yourself with a clear user-agent string and a contact URL.
• Back off on errors (429, 503) and respect Retry-After (Chapter 25).

Memory code: "Read the sign on the door, knock gently, and leave when asked."