19.6 How do you keep the search index in sync with the database?

In the library: New books arrive, old ones are withdrawn, a title gets corrected. If nobody updates the cards, the cabinet sends people to books that aren't there. The shelves are the truth; the cards are a copy that must follow them.

In the system: The database is the source of truth; the search index is a derived copy, like a cache (Chapter 12). Ways to keep it updated:

Figure: send committed changes through a log, with versions, instead of writing to both places from the app.

• Dual write (avoid): the app writes to the database, then to the search index. If the second write fails, or two updates arrive out of order, the two drift apart silently.
• Change data capture (CDC): a tool reads the database's own change log (for example, Debezium reading the replication log) and streams every committed change to an indexer.
• Outbox pattern: the app writes the change and an "outbox" event in the same database transaction (Chapter 17), and a separate process publishes the events (Chapters 22 and 23).
• Periodic batch reindex: rebuild from scratch nightly; simple, but results are hours stale. Often used as well, to repair drift.

Details that matter:

• Out-of-order events: carry a version (or the update time) with each change, and ignore anything older than what's already indexed. In the chapter's code, a late event with version 2 arriving after version 3 is skipped, so an old title doesn't come back.
• Deletes must flow through too, or deleted books stay searchable.
• Search is eventually consistent. Elasticsearch makes new documents searchable after a refresh, about once a second by default, so a just-added book may take a moment to appear. Design the UI for that (show the new item from the database right after the user creates it).

Memory code: "The shelves are the truth; the cards follow the change log."

---