Here is a puzzle. Elasticsearch's inverted index is heavily compressed and interleaved on disk; inserting one document into the middle of it would mean rewriting enormous files. Yet the machine accepts hundreds of thousands of writes per second. This essay is about the trick that makes it possible, and the live simulation below lets you run it yourself.
Here is a puzzle. The inverted index is heavily compressed and interleaved on disk; inserting one document into the middle of it would mean rewriting enormous files. Yet Elasticsearch accepts hundreds of thousands of writes per second. How?
The answer, inherited from the Lucene library at its core, is one of the great tricks of systems engineering: never modify files, only create new ones. Incoming documents accumulate in a memory buffer. On every refresh, the buffer is written out as a small, complete, immutable mini-index called a segment. A search simply asks every segment and combines answers. Deletion doesn't touch the segment either; the document is marked dead in a side file and filtered from results. And a background process constantly merges small segments into bigger ones. Below, the whole machinery is running live; you control its dials:
Every oddity of Elasticsearch falls out of this design. Why "near real-time" rather than instant? A document is searchable only after the next refresh. Why doesn't a crash lose the buffer? Every write is also appended to a plain journal, the translog, replayed on restart. Why does the machine spend idle time merging segments? Because searching a thousand tiny indexes is slower than ten large ones, and merging is when deleted documents are finally, physically, gone.
Immutability pays dividends everywhere: it is why replication and snapshots are cheap, and why the query engine can parallelize by segment.