Search asks "which documents match?" Analytics asks something else entirely: summarize a field across millions of documents. This essay is about the second question: the columnar layout that makes it cheap, the bucketing engine behind every dashboard, and the beautiful algorithm that trades a sliver of accuracy for almost all of the memory.
Search asks "which documents match?" Analytics asks something else entirely: summarize a field across millions of documents. The inverted index is the wrong tool; you'd be hopping all over the disk. This is what doc values are for, every field stored again as its own contiguous column:
On columns, Elasticsearch builds aggregations: sort documents into buckets, by time, by term, by range, and compute metrics inside each. It's a histogram engine living inside the search engine, distributed like everything else. Here are 240 log events, bucketed live:
Some summaries, though, resist this treatment. "How many distinct users?" seems to require remembering every user you've seen, unbounded memory. Elasticsearch's cardinality aggregation refuses to pay that and uses HyperLogLog instead: hash every value and remember only the most surprising hash patterns seen, in a few fixed bytes. Feed it:
This is a deliberate philosophy: at sufficient scale, a fast answer that is 99% right beats an exact answer you cannot afford. Percentiles get the same treatment. One kind of numeric data is so extreme, so relentless, that even columns weren't enough for it: metrics got their own engine.
One kind of numeric data is so extreme it outgrew even columns: metrics. That story has its own essay.