Somewhere behind almost every search box you have ever typed into, a curious machine is at work. It takes in text, chops it into pieces, files every piece away in several filing systems at once, and then answers questions in milliseconds, best answers first. This essay follows one document into Elasticsearch and one query back out. Every figure is interactive; the essay works best if you play with all of them.
Everything begins with a document: a piece of JSON. Here is a log line from an imaginary payment service. It looks like one thing, but Elasticsearch will not store it as one thing. Guided by a schema called the mapping, each field's value is copied into one or more specialized structures, each built to answer a different kind of question. Hover over the fields:
text field is analyzed into the inverted index; a keyword like service is indexed whole and also columnar; numbers, dates and IPs get range trees; vectors join the neighbor graph.This is the first big idea, and it explains half of everything downstream: Elasticsearch pays generously at write time to make read time almost free. One incoming document might be written five different ways. That sounds wasteful until you remember the asymmetry of the workload: a log line is written once and queried, aggregated, sorted, and dashboarded thousands of times.
Let's follow the message field, mapped as text. Computers are poor readers; to a computer, payment, Payment, and payments are three unrelated byte sequences. So before text is indexed it passes through analysis, an assembly line that normalizes raw text into tokens. Edit the sentence:
The stages are humble: split on punctuation, lowercase, discard glue words, and trim words to a root so payments, payment and paying collapse together. Crucially, queries pass through the same pipeline, so document and query meet in the middle, in normalized token space. Elasticsearch ships dozens of analyzers, for English, Japanese, file paths, email addresses, and lets you compose your own from tokenizers and filters, including synonym filters that make error and failure meet in the middle too.
Tokens now need to be stored so they can be found without reading everything. Elasticsearch borrows the oldest trick in publishing, the index at the back of a book: precompute, for every token, the list of documents that contain it, called a postings list. Below, four documents and the real index built from them. Try the search box, and hover the index rows:
payment timeout, then checkout error, then Payments.Watch what the query actually does: it never reads a document. It looks up tokens, walks posting lists, intersects them. The documents are touched only at the very end, for display. The cost of search was paid when the index was built.
Real questions are rarely one word. They look like: messages matching "timeout", from the checkout service, slower than two seconds. Elasticsearch composes such questions with a bool query, and it draws a distinction that shapes everything about how the machine spends its effort: some clauses score, and some merely filter.
A must clause asks "how well does this match?" and contributes to relevance. A filter clause asks a yes/no question, contributes nothing to the score, and, because yes/no answers never change for a given segment, its result is cached as a bitset and reused by every future query that asks it. Toggle the clauses:
This ordering, cheap cached filters first, expensive scoring on the survivors, is the quiet workhorse optimization of nearly every production query. The range filter, incidentally, is answered by the BKD tree from chapter one: numbers organized so that "≥ 2000" is a walk down a tree, not a scan.
Humans misspell. A search engine that returns nothing for paymnet has failed at its one job, so Elasticsearch can match terms approximately, within a budget of single-character edits, insertions, deletions, substitutions, called edit distance. Type a misspelling:
The other human habit is not finishing words. Search-as-you-type works by indexing prefixes: at write time the term payment is also indexed as p, pa, pay, and so on, the familiar bargain of paying at write time, so that each keystroke is an ordinary exact lookup:
Matching is half the job; the other half is deciding what comes first. Elasticsearch scores matches with BM25, which encodes three intuitions: repetition matters, but with diminishing returns; rare terms matter more than common ones; and matches in short documents beat matches diluted in long ones. First, the shape of "diminishing returns":
Now the whole formula against a real corpus. Type any query; the five log lines re-rank live, each bar split by term so you can see which word earned the score:
payment, then payment timeout, then cache. The rarer word earns the taller bar: rarity is worth more.Searching is only half the machine. Segments explains how all of this gets written fast, and One machine from many spreads it across computers.