Art of Codeartofcode
Lesson 5 of 15

19. Search and Inverted Indexes

1
The picture to keep in your head
2
19.1 Why doesn't `LIKE '%word%'` scale?
3
19.2 How do you tokenise and normalise text?
4
19.3 What is an inverted index?
5
19.4 How does ranking work (TF-IDF, BM25 in brief)?
6
19.5 How do prefix search and tries work?
7
19.6 How do you keep the search index in sync with the database?
8
19.7 Where does Elasticsearch fit in the big picture?
9
19.8 How do you shard a search index?
10
Memory card and where the analogy breaks
11
Practice: five levels
12
Interview questions
13
Solutions
14
Sources
15
Code lab
19. Search and Inverted Indexes

19.4 How does ranking work (TF-IDF, BM25 in brief)?

3 min read+10 XP

In the library: Ask for "databases" and get 4,000 books. The librarian puts first the books about databases, not the ones that mention the word once in passing. A book where "databases" is in the title beats one where it's buried in a 500-page appendix. And a rare, specific word like "Kerala" tells the librarian more than a common one like "book".

In the system: Ranking gives each matching document a score. Two classic ideas:

  • TF (term frequency): the more often the term appears in the document, the more relevant it probably is.
  • IDF (inverse document frequency): terms that appear in few documents are worth more. "kerala" in 50 books is a strong signal; "guide" in 300,000 is weak.

TF-IDF multiplies them: score = TF × IDF, summed over the query's terms.

BM25 is the improved version most engines use by default (Elasticsearch's default similarity is BM25, with parameters k1 = 1.2 and b = 0.75). It adds two fixes:

  • Saturation (k1): the fifth mention of a word adds much less than the first. A book can't win just by repeating "databases".
  • Length normalisation (b): a match in a short field counts for more than the same match in a long one.

TF-IDF rises in a straight line as a word repeats; BM25 levels off.

Figure: BM25 rewards a word appearing, but stops rewarding it for repeating.

In the chapter's code, a book whose short title says "databases" three times scores 3× another book under TF-IDF, but only about 1.9× under BM25.

Real search adds much more on top:

  • Field boosts: a match in the title counts more than in the description.
  • Business signals: popularity, ratings, stock, recency, personalization.
  • Learning to rank and semantic (vector) search with embeddings, for "meaning" rather than exact words. Many systems combine keyword (BM25) and vector results, known as hybrid search.

Memory code: "Rare words count more; repeating a word stops paying."