An index in the wild lives on many computers that share work, fail without warning, and still have to agree on one truth. This essay is about how Elasticsearch runs a search across a fleet, survives the loss of its members, and elects the one brain that keeps everyone honest.
Everything so far happened on one computer, but an index in the wild is sharded: cut into pieces, each a fully working index of its slice, spread across machines. Any node can receive your query and becomes its coordinator: it broadcasts to one copy of every shard, each shard ranks its own documents and returns only its few best candidates, just IDs and scores, and the coordinator merges them and fetches full text for the winners only.
The figure below holds one search, frozen in time. The timeline slider is time: drag it slowly through the request's life, backwards too. Then change the shard count and drag through it again:
The elegance is in what doesn't move: a query over a billion documents ships a few hundred bytes each way. The heavy work happens where the data lives, in parallel. Add shards and the same search gets faster, at the price of a slightly busier merge, a trade you just measured yourself.
Machines fail, routinely. So each shard keeps understudies, replicas, full copies on different nodes, which serve reads in the meantime. Index a few documents, then kill a node:
No human decides anything: the cluster notices missing heartbeats, promotes replicas, rebuilds spares on the survivors, and searches keep flowing throughout. This is how a reliable service is built from unreliable computers.
But who exactly "notices"? Who decides which replica gets promoted? A cluster of equals with no referee would tear itself apart, so Elasticsearch nodes hold roles. Data nodes hold shards and do the heavy work; any node can play coordinator for a request, as in chapter 1; ingest nodes run the ingest pipelines (see the life of a log); and a small set of master-eligible nodes can hold the one job that must never be held twice: the elected master, sole owner of the cluster state, the versioned map of every index, mapping, and shard location that every node carries a copy of, but only the master may change.
The danger is obvious: if a network hiccup splits the cluster and both halves elect a master, both halves accept writes, and the data diverges irreparably; the dreaded split-brain. The defense is arithmetic: electing a master requires a quorum, a strict majority of the master-eligible nodes, and no two halves of a split can both hold a majority. Test it:
Notice the philosophy in the failure case: a group without a majority doesn't guess, it steps down and refuses writes. Elasticsearch would rather be briefly unavailable than quietly wrong; unavailability heals, divergence doesn't.
The same thinking extends outward in rings. Replicas are placed with allocation awareness, never in the same rack or availability zone as their primary, so one flooded datacenter room doesn't take both copies. Snapshots stream every index to object storage as the last line of defense, and they're incremental, only new segments are copied, the immutability of segments paying yet another dividend. And for the largest deployments, cross-cluster replication keeps an entire second cluster in another region following the first, ready to take over, while cross-cluster search lets one query span them all.
The partial-then-merge pattern you scrubbed through here reappears inside ES|QL's compute engine, and the storage being distributed is built from immutable segments.