The cluster: elections, allocation, and disasters

The Elasticsearch essays · February 2026

One machine from many showed a healthy fleet sharing work. This essay is about everything that goes wrong: machines dying mid-write, networks splitting a cluster in half, and the arithmetic and choreography that keep the data whole through all of it.

1 · When machines fail

Machines fail, routinely. So each shard keeps understudies, replicas, full copies on different nodes, which serve reads in the meantime. Index a few documents, then kill a node:

Documents hash to a shard (solid = primary, dashed = replica). Click a node to fail it; a replica is promoted. Kill two.

No human decides anything: the cluster notices missing heartbeats, promotes replicas, rebuilds spares on the survivors, and searches keep flowing throughout. This is how a reliable service is built from unreliable computers.

One brain: the cluster state and its election

But who exactly "notices"? Who decides which replica gets promoted? A cluster of equals with no referee would tear itself apart, so Elasticsearch nodes hold roles. Data nodes hold shards and do the heavy work; any node can play coordinator for a request, as in One machine from many; ingest nodes run the ingest pipelines (see the life of a log); and a small set of master-eligible nodes can hold the one job that must never be held twice: the elected master, sole owner of the cluster state, the versioned map of every index, mapping, and shard location that every node carries a copy of, but only the master may change.

The danger is obvious: if a network hiccup splits the cluster and both halves elect a master, both halves accept writes, and the data diverges irreparably; the dreaded split-brain. The defense is arithmetic: electing a master requires a quorum, a strict majority of the master-eligible nodes, and no two halves of a split can both hold a majority. Test it:

master-eligible nodes3

network partitionnone

Click nodes to kill and revive them; drag the partition slider to cut the network. The crown only ever sits in a group holding a majority.
The experiments: with 3 nodes, kill the master and watch a survivor take over; then kill another, and the last node refuses to rule alone. Set 5 and partition 3|2: the majority side carries on, the minority side cannot elect anyone. Now set 4 and partition 2|2. A naive majority rule would deadlock here; Elasticsearch avoids it by keeping only three of the four in its voting configuration, so one side still holds two of three votes. Notice what the fourth node does not buy you: like three, four survives the loss of exactly one master-eligible node. That is why clusters run three or five.

Notice the philosophy in the failure case: a group without a majority doesn't guess, it steps down and refuses writes. Elasticsearch would rather be briefly unavailable than quietly wrong; unavailability heals, divergence doesn't.

The same thinking extends outward in rings. Replicas are placed with allocation awareness, never in the same rack or availability zone as their primary, so one flooded datacenter room doesn't take both copies. And for the largest deployments, cross-cluster replication keeps an entire second cluster in another region following the first, ready to take over, while cross-cluster search lets one query span them all.

2 · Making room: allocation and rebalancing

Failure is the dramatic case; the everyday case is change. You add a node, and the cluster should use it; an index grows hot, and its shards should spread. The master continuously solves this placement puzzle under hard rules (never a replica beside its primary; respect zone awareness) and soft goals (balance shard counts and disk). Resize the cluster:

nodes3

Twelve shards (six primaries, six replicas) redistributing as the cluster grows and shrinks. Watch the invariant hold at every size: a primary (solid) and its replica (outlined, same number) never share a node. At two nodes the layout is forced; at six there is slack, and shard counts even out. In real clusters the same solver also weighs disk usage, zones, and per-index limits.

3 · Rebuilding a copy

Promotion is instant, but it leaves the cluster one copy short, and the replacement replica is not conjured, it is recovered: built from the primary while writes continue to flow. The process has two phases, and knowing them explains most recovery dashboards you will ever stare at:

–

phase

–

transferred

recovery timeline

Phase one copies the primary's immutable segment files, gigabytes of them, and immutability is what makes this safe: the files cannot change mid-copy. But writes keep arriving during the copy, so the primary keeps the history of every operation since the copy began (a retention lease stops that history from being cleaned up), and phase two replays those operations onto the new replica, a much smaller catch-up. When the replay gap closes, the copy is marked in-sync and starts serving reads. A recovering shard that never leaves phase two is a shard being written to faster than it can catch up. And when a node only restarted, it still holds almost all of the files: recovery can then skip phase one and replay just the operations it missed.

4 · Surviving a whole zone

Nodes fail one at a time; datacenters fail all at once. If both copies of a shard happen to sit in the same availability zone, the replica bought you nothing the day the zone goes dark. Allocation awareness teaches the allocator geography:

–

data status

zone-aware allocation zone B fails

With awareness on, every primary's replica lives in the other zone, so killing zone B leaves every shard with a surviving copy: degraded, but whole. Turn awareness off and the allocator optimizes for balance alone; shards 2 and 3 end up with both copies in zone B, and when it fails, they are simply gone: red, unsearchable, unrecoverable until the zone returns. The checkbox is one cluster setting; the difference is your data.

5 · Backups that share

The last line of defense is the snapshot: a copy of the data in object storage, outside the cluster's fate entirely. Snapshots here have a superpower inherited, once again, from immutability: since segment files never change, snapshot two does not re-upload what snapshot one already stored:

–

naive: 3 full copies

–

actually uploaded

data changed between snapshots

Three nightly snapshots of a 100 GB index. Blue blocks are segments uploaded by that snapshot; gray blocks are references to files a previous snapshot already uploaded. On a typical logging index where old segments never change, a nightly snapshot uploads only the day's churn, which is why snapshotting terabytes nightly is routine rather than heroic. The sharing works at the level of whole files, so merges matter: when small segments merge into a big one, the big one is a new file and gets uploaded, which is why an index that is still being written and merged re-uploads more than a sealed one. Deleting an old snapshot only frees the files no other snapshot still references.

Notice how often the answer was immutability: recovery copies files that cannot change, snapshots share them, and the rest of the machine leans on the same trick. Related: the immutable segments that make replication and snapshots cheap, and the search path all this reliability exists to protect.