Finding every matching document is the easy half of search. A query against production logs might match fifty thousand lines; the hard half is deciding what comes first. This is where a search engine parts ways with a database: it doesn't return matches, it returns matches in order of relevance. This essay is about how that order is computed, and how you bend it.
Elasticsearch scores matches with BM25, which encodes three intuitions: repetition matters, but with diminishing returns; rare terms matter more than common ones; and matches in short documents beat matches diluted in long ones. First, the shape of "diminishing returns":
Now the whole formula against a real corpus. Type any query; the five log lines re-rank live, each bar split by term so you can see which word earned the score:
payment, then payment timeout, then cache. The rarer word earns the taller bar: rarity is worth more.Of BM25's three intuitions, rarity does the most work, so it deserves its own dials. Inverse document frequency asks one question: out of all the documents, how many contain this term? Drag both answers:
Every score Elasticsearch produces can be itemized. Ask for an explanation and the response is a tree: the final score at the root, and underneath it, every factor from the last two sections with its exact value. This one is computed live from a six-line corpus; click any row to unfold it:
_explain API's answer. The root sums one branch per query term; each branch multiplies idf (from chapter two) by the saturation term (from chapter one); the leaves hold the raw ingredients: term frequency, document frequency, this document's length against the average. When a ranking surprises you in production, this tree is where the surprise stops being one.Real documents have structure: a title, a body, maybe tags. A match in a title usually means more than the same match buried in a body, and Elasticsearch lets you say so with boosts, per-field multipliers on the score. Tune them:
timeout. Each document's bar splits into its title contribution (dark) and body contribution (light). Slide title weight to 0 and the runbook whose body mentions timeouts constantly takes the lead; slide it up and the incident titled "Timeout" wins. Boosts don't change what matches; they change what wins.Boosting weighs fields; there is a prior question of how to combine them at all. When two words can land in two different fields, three strategies give three different podiums. Same query, same documents:
Relevance has a time dimension the words can't see: this morning's incident ticket may matter more than a better-written postmortem from last year. Elasticsearch handles this with decay functions: the text score is multiplied by a curve that falls with age, and you choose how fast. The choice decides who wins:
exp decay on the date field with scale = the half-life and decay: 0.5. The curve is the policy: how fast should the past fade?These are the honest levers of first-page quality: the formula's own dials (k₁, b), field boosts, and time decay. There is one more, the most powerful of all: bringing in a second judge entirely. That story, vectors and rank fusion, lives in The vector database.
One secret remains. Everything above prices a single document, but a query wants the top ten of ten million, and Elasticsearch does not score ten million documents to find them. Each block of a posting list stores its maximum possible score, and once the tenth-best result so far beats a block's maximum, that whole block is skipped unread. The trick is called block-max WAND, and you can watch it win, and watch it lose when you ask for more results:
track_total_hits), and track_total_hits: true forbids it altogether, which is why exact totals are slower.That completes the pricing machinery: what a document is worth, why rarity dominates, how fields and freshness bend it, and how first is found without scoring the rest. The structures these shortcuts live in are the subject of Inside the inverted index.