Search asks "which documents match?" Analytics asks something else entirely: summarize a field across millions of documents. This essay is about the second question: the columnar layout that makes it cheap, the bucketing engine behind every dashboard, and the beautiful algorithm that trades a sliver of accuracy for almost all of the memory.
The inverted index is the wrong tool for that: it maps terms to documents, so reading one field across millions of documents would mean hopping all over the disk. This is what doc values are for, every field stored again as its own contiguous column:
On columns, Elasticsearch builds aggregations: sort documents into buckets, by time, by term, by range, and compute metrics inside each. It's a histogram engine living inside the search engine, distributed like everything else. Here are 240 log events, bucketed live:
Some summaries, though, resist this treatment. "How many distinct users?" seems to require remembering every user you've seen, unbounded memory. Elasticsearch's cardinality aggregation refuses to pay that and uses HyperLogLog instead: hash every value and remember only the most surprising hash patterns seen, in a small, fixed amount of memory. Feed it:
precision_threshold × 8 bytes (24 KB at the default 3,000), counting near-exactly up to the threshold and within a few percent beyond it.Percentiles get the same treatment as counting. "What is the p99 latency?" seems to require keeping every value and sorting; the t-digest keeps a bounded set of centroids instead: small, precise clusters near the tails where precision matters, big coarse ones in the middle where it doesn't. One knob trades memory for accuracy:
This is a deliberate philosophy: at sufficient scale, a fast answer that is 99% right beats an exact answer you cannot afford.
Every aggregation so far counts what is common. But "common" is usually boring: the most frequent user-agent among your errors is Chrome, because Chrome is the most frequent user-agent, period. The interesting question is what is common in your errors compared to everything else. That is the significant-terms aggregation, and it is a different animal entirely:
The approximations so far, HyperLogLog and t-digest, are baked into their aggregations. There is also a knob that approximates any aggregation: run it on a random sample of matching documents and scale up. The bargain is lopsided in your favor: the speedup grows with the sampling rate, the error only with its square root:
random_sampler aggregation accepts probabilities below 50%, or exactly 100%; the speedup is close to linear on large datasets.)One kind of numeric data is so extreme, so relentless, that it outgrew even columns: metrics. That story has its own essay.