A metric data point is the smallest document imaginable: a metric name, a few identifying labels (which host, which service), a timestamp, a number. Perhaps twenty bytes of actual information, arriving millions of times per second, forever. Stored as an ordinary document, each point drags along machinery built for documents: an _id, sequence numbers for concurrency control, index entries per field. The wrapping weighs more than the gift; historically, about 25 bytes per point. This essay walks the rebuild that fixed it, and every figure is an experiment you run yourself. (The columnar foundations live in Numbers at scale.)
Elasticsearch's time-series mode (TSDS) attacks that overhead with five changes, and the figure below lets you apply them one at a time, in any order. Each colored square is one data point from one of three series; the bar underneath is where its bytes actually go:
Walk the toggles and you've walked the engineering. Sort by series, then time, and route each series to a single shard, so every series is one contiguous, sequential run on disk; nothing else works without this. Then the big one: for timestamps and dimensions, lightweight per-block skip summaries replace the inverted index and BKD tree entirely, deleting the duplicated per-field index machinery, about ten of the twenty-five bytes, while the skip-driven queries later in this essay stay fast. The _id becomes synthetic, computed as _tsid plus timestamp instead of stored and indexed (a bloom filter keeps get-by-id working), five more bytes. That _tsid deserves a proper introduction, because the whole engine leans on it: it is the identity of one time series, a compact hash of the series' stable dimension set. Points share a _tsid while their timestamps and values change; alter a single identifying dimension and you are, by definition, in a different series. Sequence numbers are trimmed because metrics never see concurrent updates: four bytes of replication metadata gone. And the numeric codec blocks widened from 128 to 512 values, so repeating dimension patterns, IPs and MAC addresses cycling host by host, finally fit inside the compressor's window. Together: 25 bytes become 3.75, and a day of a billion points shrinks from 25 GB to under 4.
Queries got the mirror treatment. Because the data is sorted by series and time, each series is a run of compressed blocks, and each block carries a tiny label: which series, which time span. A metrics query almost always begins with where and when, "payments, last four hours", and both conditions are answered by reading labels, not data. Blocks that can't match are skipped before they are ever decompressed. Steer the query yourself:
What happens to the surviving blocks is just as deliberate. They decode zero-copy, straight into the primitive-typed blocks the compute engine works on, no intermediate row objects, no allocator churn; a dimension that repeats for a thousand consecutive points isn't even expanded, it becomes a constant block, one value and a count. The work parallelizes by slicing the _tsid range by prefix, so every thread still reads its slice of disk in order. Even the humble act of rounding timestamps into buckets was rebuilt: when the bucketing is timezone-agnostic, it collapses to a modulo, worth a third of the response time on some queries. None of these is a headline; together they are the headline.
The most important question you can ask of a counter, how fast is it growing?, hides a trap. Counters only go up, until the process restarts and the counter falls back to zero. Subtract naively across a restart and you compute a huge negative rate; a dashboard built on that lies to you at exactly the moments that matter, the restarts. Elasticsearch's rate() walks each series in time order (the sort again) and treats every drop as a reset, adding the lost baseline back:
This is what the TS command in ES|QL is for: it computes functions like rate() inside each series first, and only then aggregates across series. That command, and the fact that PromQL itself now runs natively against this storage, deserve their own essay: The language of time.
None of this matters if the data can't get in fast enough, so the write path was rebuilt around the same shape. Toggle the five optimizations and watch the cost of accepting one million points per second:
_tsid so data nodes never repeat the work; the recovery copy of the source stops being written and gets re-synthesized on demand instead; and segments carry lightweight skippers instead of hot inverted structures, so the eternal background merging gets cheaper too. Every core you extinguish is a real machine somebody doesn't have to run.Metrics age faster than any other signal: nobody needs ten-second resolution on last quarter's CPU. So aging metric indices get downsampled: raw points are replaced by one summary document per series per interval, keeping the min, max, sum and count exactly (for gauges; counters keep their last value), and letting everything else go. Choose what history becomes:
Everything above assumed you know how many time series you have. Nobody does, because series count is a product, not a sum: every distinct combination of dimension values is its own series, its own _tsid, its own storage and memory. Arm the bomb yourself:
endpoint, with 200 values, and you didn't add 200 series, you multiplied by 200: 160,000 series, each one indexed, sorted and tracked. The classic accident is a label that was never meant to be a dimension at all, a request id, a container hash, and it is how monitoring systems die. The defense is choosing dimensions deliberately: dimensions are for things you group by; everything else belongs in the point, not the key.And since this essay makes performance claims, the published benchmark numbers (Elasticsearch 9.x, default configurations, open workload generator, single node):
| query type | vs Prometheus/Mimir | vs ClickHouse |
|---|---|---|
| gauge average | up to 30× faster | up to 8× faster |
| counter rate | up to 30× faster | up to 3.5× faster |
| storage per point | 2.5× smaller | 2× smaller |
One more number floats around this work, and it deserves its own baseline, kept deliberately separate from the table: up to 160× faster than the same metrics stored the old way, as ordinary documents in Elasticsearch's own document store. That comparison isolates the architectural value of the purpose-built path itself, engine against engine on the same data, rather than system against system; collapse the two baselines into one claim and both numbers stop meaning anything.
One last elegance: PromQL, the language most of the world's dashboards already speak, runs against these very same column files. Same storage, two dialects; nobody rewrites their dashboards to switch engines.