The language of time told the usage story: why counters lie under ordinary aggregation, and what it feels like to point a Grafana dashboard at Elasticsearch and have it simply work. This essay is the other side of the wall. The engineering team published their own account, How we built PromQL into Elasticsearch, listing five modifications they had to make deep inside the compute engine. Here we take those five apart, one interactive figure at a time.
Elasticsearch 9.5 runs PromQL natively. Not in a plugin, not in a sidecar process pretending to be a Prometheus: PromQL is another front end to the same columnar compute engine that runs ES|QL. A PromQL expression is planned into the same kind of query plan an ES|QL query becomes, then runs with the same distributed execution, the same testing infrastructure, the same release process. When the engine gets faster, both languages get faster.
The bet behind this is simple. Your metrics already live in Elasticsearch via remote write; the world's dashboards, alerts, and muscle memory already speak PromQL. Connecting the two means speaking Prometheus's language with Prometheus's exact semantics, inside an engine that was built for a different grammar. A query that returns numbers 2% off from what the same query returned last week, on the same data, is not a compatibility story: it is a silent incident.
And the hard part was never parsing PromQL; the grammar is small and well documented. The hard part was time itself: where evaluation points fall on the clock, which direction a window looks, and what the engine is allowed to know before the data arrives. Three problems, five engine modifications. Let's walk them.
ES|QL buckets time with TBUCKET, which truncates every timestamp to a wall-clock boundary: 5-minute buckets snap to :00, :05, :10, regardless of when you asked. For a human scanning a dashboard this is a feature. For PromQL it is a broken contract, because Prometheus promises evaluation at exactly start, start+step, start+2*step: a grid anchored to the query, not to the clock. The first engine modification, TSTEP, builds that grid. Drag the query start and watch the two philosophies disagree:
The divergence above is not an edge case; it is every migrated dashboard, most of the time, because real query starts are almost never multiples of the step. Wall-clock truncation would silently change every number on the chart, and the person staring at it would have no way to know. TSTEP preserves Prometheus timestamp semantics exactly: same grid, same windows, same digits.
The second modification is about direction. When PromQL writes rate(http_requests_total[5m]), the [5m] means the interval (T-5m, T]: five minutes ending at the evaluation time T, open at the far end, closed at T itself. The window looks backward, because at time T the only samples that exist are the ones already written. The engine's window evaluation had grown up around forward-facing bucket spans, so it was rewritten to run in both directions. Flip it yourself:
Notice what the toggle really shows. Both windows are five minutes wide, both touch T, and they still disagree, because traffic was doing different things before T than after. Any engine that quietly evaluated [5m] forward would produce plausible-looking charts shifted a whole window early: every point would describe the five minutes after its timestamp, so each spike shows up on the chart before it happened. Plausible and wrong is the worst failure mode in observability.
The third modification is about what a query planner can know. PromQL groups series two ways: by(job, region) keeps exactly the labels you name, and without(instance, pod) keeps everything except the ones you name. The first is a gift to the planner: the output schema is right there in the query. The second is open-ended: the set of remaining labels depends on which label sets actually show up in the data, so the engine cannot pre-plan a fixed schema. It has to discover the grouping dimensions at runtime. Build both clauses against a small fleet:
That byte strip at the bottom is the fourth modification, dimension packing. Metric label sets are wide: a group key of five or six string labels is normal, and dragging every one of them through the aggregation as a separate column costs real memory and real time. So the engine encodes the whole discovered key into a compact 16-byte representation up front, and aggregates on that, cutting the per-column overhead exactly where label-heavy queries hurt most.
The fifth modification sits at the far end of the pipeline. The engine thinks in columns; a Prometheus client expects series, one object per label set with its samples on the step grid. The first implementation built that answer out of maps, lists and boxed numbers: a little object for every value, millions of times. The TimeSeriesCollapse operator instead groups rows by series and aligns their samples to the step grid inside the pipeline, keeping the compact columnar blocks the whole way, so the HTTP layer can serialize those blocks directly and the per-value allocations never happen.
How do you know a second implementation of someone else's language actually speaks it? You measure. The team harvested a corpus of more than 2,000 real PromQL queries from public repositories and classified every one by the features it uses. That corpus, not intuition, decided what to build first:
Against that corpus, more than 80% of the queries run on Elasticsearch without modification. The answers are checked three ways: the Prometheus compliance tests; differential testing, running the same queries against a real Prometheus and comparing results; and fuzz testing, generating queries no human would write to shake out the corners. And the remaining fifth is stated plainly: on() and group_left vector matching, topk, sort and their relatives, label_replace and label_join, the @ modifier, and mixed-offset expressions are still on the list.
What shipped GA in 9.5 is already the working core of the ecosystem: remote write in, range and instant queries out over the HTTP API, the metadata and build-info endpoints, native histogram functions (histogram_quantile, histogram_count, histogram_sum), per-selector offsets like [5m] offset 1h, and a top-level or of up to eight operands. Enough that Grafana treats Elasticsearch as an ordinary Prometheus data source and most dashboards just render.
Step back and the shape of the thing is the point. One engine now answers ES|QL and PromQL from the same columns: two grammars, one planner, one set of bytes on disk. How those bytes got so small is the metrics engine; the machinery both languages ride is ES|QL: one language for the whole machine. The clock problem is solved the only way it could be: not by translating Prometheus, but by teaching the engine to keep Prometheus's time.