Semantic search in practice

The Elasticsearch essays · March 2026

The vector database explained how Elasticsearch stores and navigates vectors. This essay is about everything around that: where embeddings come from, the two very different shapes they take, what to do when documents are too long to embed whole, and the heavier models that polish the final ranking. Map a field as semantic_text and Elasticsearch runs all of it for you; this essay is what "all of it" means.

1 · Sparse: a bag of weighted ideas

Elastic's own ELSER model produces sparse vectors: instead of coordinates in an abstract space, it expands text into weighted terms, including terms that never appear in the text, the ideas it implies. The expansion is interpretable, you can read it, and it is served by the ordinary inverted index. Pick an input, and see how it finds a document it shares no words with:

query expansion

document:

–

words typed in common

–

match score · Σ query weight × doc weight

–

query terms looked up

prune terms below weight0.00

Illustrative expansions in the style ELSER produces (the real model emits dozens to hundreds of weighted terms). Chip size is weight; purple chips are terms the query and the document both expanded to. The query and the document share no typed word, yet they match through implied ones like cart and purchase, in the plain old inverted index: meaning, served by the machinery of words. Every expanded term is one more posting-list lookup, which is why Elasticsearch can prune low-weight terms; drag the slider to see how much of the score survives.

2 · Chunking: long documents, small windows

Embedding models read a bounded window of text; a hundred-page manual doesn't fit, and even when text fits, one vector for a whole document blurs all its topics together. So semantic_text automatically chunks long fields: each chunk embedded separately, each searchable on its own. By default it groups whole sentences into chunks of up to 250 words, overlapping by one sentence. Chunk size is a real trade, and you can feel it:

query: "why did the payment retry storm happen?"

–

best chunk's match strength

–

of the answer inside it

–

chunks (vectors to store)

chunk size30 words

One incident report, split into chunks; the greener a chunk, the better it matches the query, and the outlined one wins. The answer is one sentence: the real cause was a retry storm…. Tiny chunks score high but cut that sentence in pieces, so the winning chunk holds a fragment of the answer; huge chunks hold it whole but dilute it into everything around it: the match weakens, and at some sizes a chunk that merely repeats the query's words more often wins instead. Watch the second readout jump as the boundaries move. The craft is the middle, and sentence boundaries help: they never cut an answer mid-thought.

3 · Late interaction: token meets token

A single vector per chunk is still a summary, and summaries blur. A higher-precision technique keeps one vector per token and compares query and document token-by-token: each query token finds its best match in the document, and the score is the sum of those best matches, maxSim. Elasticsearch stores such vectors in the rank_vectors field. Compare two documents against the same query:

–

maxSim score

Rows are query tokens, columns are document tokens; each cell is a similarity, and the ringed cell in each row is that query token's best match. Both documents contain "payment" and "retry" words, but doc A's tokens agree with the query's intent across the board, and the sum shows it. This is why late interaction out-ranks single-vector search when precision matters, at the price of storing a vector per token.

4 · The final judge: rerankers

Everything so far runs at retrieval scale, over millions of documents. The last refinement runs on the podium only: a reranker, a cross-encoder that reads query and candidate together, token interactions and all, and re-orders the top handful with far more care than any first pass could afford. Since acquiring Jina AI, Elastic serves Jina's multilingual rerankers and embedding models through its inference service, alongside ELSER. Cheap judges nominate; an expensive judge decides; the same economics as everything else in this machine.

query: "why did checkout payments fail last night"

–

ranking quality (NDCG vs ideal)

–

model reads per query

rerank the topoff

Retrieval scored each document against the query separately, and it ranks a keyword-heavy postmortem from March highly while burying last night's actual incident report at position seven. The reranker reads query and document together, catches that "last night" matters and that March doesn't, and rewrites the podium. But it can only reorder what it is shown: rerank the top 5 and the incident report, sitting at 7, never reaches it. The cost is real, one model inference per candidate, which is why the window is the whole decision.

How the lexical and semantic candidate lists get merged before the reranker sees them, reciprocal rank fusion, lives in The vector database.

5 · The budget: three k's

Assembled, a semantic pipeline is a funnel of three stages, each smarter and more expensive per document than the last, so each sees fewer documents: retrieve broadly, fuse the lists, rerank the survivors. Each stage has a k, and the three k's are the entire economics of the system. Tune them:

–

answer quality (proxy)

–

estimated latency

retrieve per retriever

keep after fusion

rerank

Retrieval is cheap per document, so a wide first net costs little and misses less; the reranker is about a hundred times heavier per document here, so its k quickly dominates latency. Watch the quality meter's shape: it climbs steeply as each starved stage is fed, then flattens hard. Every production RAG system converges to the same posture: generous retrieval, moderate fusion, a stingy podium.