ES|QL: one language for the whole machine

The Elasticsearch essays · July 2026

Elasticsearch speaks several dialects, but its newest and most expressive voice is ES|QL, a piped language in which each stage transforms a table. The readable syntax is the visible tip of an entire query engine, and this essay descends through all of it: what the planner does to your pipes, and how pages of columnar blocks flow through operators at execution time.

The classic interface is the Query DSL, JSON in, JSON out, and specialized dialects exist for security event sequences (EQL), BI tools (SQL), and Kibana's search bar (KQL). ES|QL is the one meant to unify them: a single pipe syntax for search, analytics and, increasingly, metrics. Build the query stage by stage:

FROM logs-payments | WHERE status_code >= 500 | STATS errors = COUNT(*), p95 = PERCENTILE(duration_ms, 95) BY service | SORT errors DESC | LIMIT 2

Click the grayed-out stages to switch them on, in order, and watch the table transform: filter, then aggregate, then sort, then trim. Each pipe hands a table to the next.

Joining: LOOKUP JOIN

For most of its life, Elasticsearch answered "we don't do joins" with a shrug and a denormalization guide. ES|QL changed that with LOOKUP JOIN: a left join against a special lookup index, small reference data like service ownership, host inventory, threat lists, joined at query time instead of copied onto every document at write time. Toggle it:

FROM logs | LOOKUP JOIN services_lookup ON service

apply the join

The lookup index maps each service to its owning team and tier. Flip the join on: the columns appear on every row, matched by key, and rows with no match (the mystery cron job) get nulls, a left join's honest answer. The reference data lives in one place and is joined fresh, so when a team changes owners you update one document, not a billion.

ES|QL keeps growing organs like this, and the next two sections take apart the most consequential pair; beyond them, cross-cluster execution lets one query span regions. The pipe syntax stays small; the planner underneath keeps getting new moves.

Two ways to join, priced differently

LOOKUP JOIN happens at query time; its older sibling, the enrich processor, does the same join at ingest time, stamping the owner onto every document as it arrives. Same result, opposite bill:

What STATS costs: the hash table nobody sees

Every STATS … BY key builds the same thing underneath: a hash table from group key to running aggregate, filled as pages stream through. Its size doesn't depend on how many rows you scan; it depends on how many distinct keys they contain, and that is a number people misjudge by orders of magnitude:

–

hash table entries

–

memory · per node, then merged

group by

A billion rows grouped by status code is a 5-entry hash table: microscopic, however large the data. Slide to user ids and the same query wants 40 million entries per node. This is the invisible line between "STATS is free" and a circuit-breaker exception, and it's why the planner pushes the partial aggregation to the data nodes: a billion rows cross no network; only the hash tables do.

FORK: one query, several answers

The newest structural trick: FORK splits a pipeline mid-stream: up to eight branches run different continuations on the same upstream rows, and their results come back as one table, each row tagged with the branch it came from:

FROM logs | WHERE service == "checkout"
| FORK ( WHERE level == "error" | STATS errors = COUNT(*) BY code )
( STATS p95 = PERCENTILE(latency, 95) BY endpoint )

one query with FORK

Both branches start from the same filtered checkout logs and return one table, with a _fork column saying which branch produced each row. Untick the box for the alternative: two queries, two round trips, the shared filter written twice, two result sets to stitch together in the client. FORK is also the backbone of hybrid search in ES|QL: a lexical branch and a semantic branch, then FUSE to merge the rankings.

What the machine does with your pipes

That readable pipe syntax is the visible tip of an entire query engine. Between your keystrokes and the answer lies an assembly line: the query is parsed into a tree, analyzed against the mappings so every field resolves to a real, typed column, optimized by dozens of rewrite rules, and finally split into the piece that ships to every data node and the piece that stays behind to merge. Step through the four phases and watch what happens to one real query; green marks what each phase changed:

FROM retro_arcade | WHERE country == "gb" | STATS sales = SUM(price * 0.75) BY year = BUCKET(date, 1 year)

Step through the four phases: the parser only knows syntax (every ? is an unresolved name); the analyzer binds names to typed columns and quietly adds a LIMIT 1000 safety net; the optimizer folds constants and hoists the arithmetic out of the aggregation; the physical planner cuts the tree in two at the exchange. Green marks what each phase changed. Errors are caught here, before execution; at runtime a bad value becomes null and a warning, never a crash.

Notice the deepest cut, in the last phase: the aggregation splits into a partial half that runs on every data node and a final half at the coordinator, the exact same partial-then-merge pattern from distributed search. And each data node re-optimizes its own fragment locally: a node whose indices don't even have the field can answer with a shrug of nulls without reading anything.

The compute engine: pages, blocks, drivers

Execution belongs to a purpose-built engine that never processes rows one at a time. Data moves in pages, batches of rows, where each column of the page is an immutable block pulled straight from the doc-values columns. Chains of operators, one small machine each for reading, filtering, computing, aggregating, transform the pages, and a driver runs each chain, typically one driver per segment (or per slice of one), so the parallelism of segments becomes the parallelism of your query. The block shape pays one more dividend: inside an operator, the hot loops run over contiguous arrays of primitives, exactly the pattern the JVM knows how to auto-vectorize, so a single SIMD instruction processes several values at once. The layout gives the CPU predictable work, and each instruction does more of it. Run it:

timeline

parallel drivers3

rows per page4

Each driver pulls pages from its segment through the operator chain: read columns, filter (rows visibly drop), compute the new column (it appears), then fold into a partial sum. Partials stream to the coordinator, which merges them into the final answer.
The experiments: drop to 1 driver and the same 24 rows take three times as long, exactly the segments story. Change the page size and watch the trade: big pages amortize overhead, small pages start streaming results sooner.

The drivers here parallelize over segments; the partial-then-final aggregation is distributed search's oldest trick; the blocks are doc values in flight.