19.8 How do you shard a search index?

In the library: One cabinet can't hold cards for 2 million books, and one librarian can't answer every question. Split the books across three rooms, each with its own cabinet for its own books. A question goes to all three rooms; each sends back its best few; you pick the best overall.

In the system: Two ways to split an inverted index:

• By document (the common way): each shard holds all terms for a subset of documents. Every query goes to every shard (scatter-gather, Chapter 15); each returns its top 10; a coordinator merges them into the overall top 10. Elasticsearch works this way.
• By term: each shard holds some terms for all documents ("a" to "m" on shard 1). A one-word query hits one shard, but multi-word queries need several shards and huge posting lists for common words land on one machine. Rarely used in practice.

Figure: every shard searches its own books; the coordinator merges.

Practical points:

• Replicas of each shard spread the read load: more replicas, more queries per second.
• Shard count is hard to change later. In Elasticsearch, the number of primary shards is fixed when an index is created; changing it means splitting, shrinking or reindexing. Plan for growth, but don't create hundreds of tiny shards either; each one has overhead.
• Scores across shards: each shard computes IDF from its own documents, so scores can differ slightly between shards. With enough documents per shard it evens out.
• Deep pagination is expensive: page 1,000 of results means every shard returns its top 10,000 for merging. Limit page depth or use cursor-style "search after".
• Time-based indexes (one index per day, common for logs) make deleting old data cheap: drop the whole index.

Memory code: "Ask every room for its best, then pick the best of the best."

---