For fifteen years, Elasticsearch has treated every document you send it with a kind of extravagant hospitality: it keeps the original whole, tokenizes every text field into an inverted index, and lays every field out a third time as a column. Three parallel structures for one log line. This essay is about the moment that hospitality became a bill, and about columnar mode, the storage engine described in Elastic’s columnar storage article, which keeps log data once and makes everything else, the document included, something computed on demand. But to see why storing once is the right move for logs, and why it took fifteen years to make, we have to start by defending the three copies. They were never a mistake.
A datastore is not a place where data sleeps. It is a machine for answering questions, and the structures it builds decide which questions are cheap. Consider three questions you might ask of the same small pile of log lines. Find the lines that mention a timeout. Count the errors by service. Show me document #4812, exactly as it arrived. These read like siblings. To a storage engine, they are three physically different problems, and no single layout of bytes on disk makes all three of them fast.
The document store answers each with a dedicated structure. The inverted index answers the text question: it maps every term to the list of documents containing it, so “timeout” jumps straight to its postings without opening a single document (that machinery is an essay of its own). Doc values answer the counting question: every field’s values laid out again as a column, so an aggregation reads one thin stripe of the data instead of every record (why that matters at scale). And the stored _source answers the retrieval question: the original JSON, byte for byte, because a search engine that cannot hand back what you gave it is a card catalog without a library.
Notice the philosophy: nothing is computed at query time that can be precomputed at write time, and every family of questions gets a purpose-built answer. For a search workload this is exactly right. Search means rich documents in the millions rather than the trillions, every read on a human’s critical path, and one round trip that must match, rank, highlight and return. Redundancy is not the waste in that design; redundancy is the design. Fire the questions yourself and watch who answers:
Then the workload changed underneath the design. Logs and observability data arrive in terabytes per day, are written once, never updated, and read almost entirely through an analytical lens: dashboards, error-rate counts, latency percentiles, the occasional needle-in-haystack grep. And they are kept for months, because the one incident that matters will always be three weeks in the past. Under that stream, the inverted index over forty fields nobody greps and the stored copy of a trillion lines nobody fetches are still there, still costing every byte they always did, answering almost nothing. The document store was never wrong. The questions changed, and the structures did not.
Idle structures would merely be embarrassing if storage were the only bill. The deeper cost shows up on every single query, and to see it you have to look at what an analytical query actually touches. A typical dashboard panel asks something like: error rate by service, last hour. That is three fields: @timestamp, service.name, log.level. Meanwhile the mapping of a real log index, once the ECS fields, the Kubernetes labels, the cloud metadata and the tracing IDs pile in, routinely carries fifty or more fields per line. Analytical queries touch around three of them at a time.
Now the first-principles part. Storage, at every level of the hierarchy, is a byte-range machine: disks hand you blocks, the page cache hands you pages, the CPU cache hands you cache lines. You pay for every byte you drag through that pipeline, not for the bytes you actually use. Which means the only thing that matters is which values are neighbors. A row layout makes the fields of one record neighbors, which is perfect for “fetch document #4812” and terrible for “sweep one field of everything”: the three values you want are marooned inside fifty-field records, and the machine must drag entire records through the pipeline to deliver slivers of them. A columnar layout makes the values of one field neighbors, and the same query touches three contiguous strips of the file and nothing else:
error rate by service, last hour, over one million log lines. Blue is what the query needs; everything tinted is what each layout forces through the disk and the caches to deliver it. Slide the schema wider: the saving crosses 90% around 30 fields, and real log mappings sail past 50.This is the arithmetic behind the headline claim: on wide datasets, columnar layouts cut the I/O of analytical queries by 90% and more, not through any cleverness, but because reading three fields finally costs three fields. And note what the figure quietly concedes: for fetching one whole document, the row layout is the better shape. Layouts are not better or worse; they are bets on which questions will come.
The second win is quieter and, over months of retention, larger. Compression is prediction: a compressor earns its keep by guessing the next byte from the bytes it has seen, and it pays out exactly as well as the data is predictable. Run a general-purpose compressor over row-ordered log records and it sees a timestamp, then an enum, then free text, then a number, then a fresh timestamp: the signal keeps changing dialect mid-sentence, and the compressor settles for the 1.5–3× that row stores typically achieve.
Turn the data ninety degrees and every block becomes monolingual. A column of timestamps is a nearly-arithmetic sequence; a column of service names is the same dozen strings on repeat; a column of durations shares a distribution. Homogeneity is predictability, and better: once the engine knows a whole block is timestamps, it can stop using a general-purpose compressor and use a codec built for timestamps. Delta encoding for the clock, dictionary plus run-length for the enums, and an honest general-purpose fallback for the one column that really is prose. Columnar stores land at 5–10× routinely, and on low-cardinality fields at 20–30×, which is not a bigger discount but a different economic regime: the same disks now hold months where they held weeks. Pick a field and watch its codec work:
The slider is the whole lesson in one control. A value compresses according to how few of it there are: three service names cost two bits each once the dictionary knows them, while two thousand cost eleven bits plus a dictionary that has itself become data. This is why the columnar literature is obsessed with cardinality, why message gets the humble general-purpose treatment, and why schema design quietly matters again: keep enumerable things enumerable, and the storage engine will pay you back thirtyfold.
Smaller bytes are only half of query speed. The other half is a two-verb discipline that columnar layouts make possible: skip what cannot match, then sweep what might. Each column is cut into blocks, and each block carries a scrap of metadata: the minimum, maximum and count of what is inside. A filter reads those three numbers first. If a block’s time range misses the query window, or its count for ERROR is zero, the block is rejected before decompression, thousands of values dismissed for the price of reading three numbers. The metrics engine proved this discipline on time series; the same trick now runs on logs.
The blocks that survive get swept, and here the columnar layout pays a second time, in a currency the previous sections have not touched: CPU shape. A decoded block is a long, regular array, which is precisely the food modern processors are built to eat. Instead of the classic per-document loop, fetch a value, compare, branch, advance, a vectorized operator runs one tight comparison over a batch of values at a time, keeping the pipeline full and letting SIMD units compare many values per instruction. The SIGMOD literature the article cites puts the advantage at one to three orders of magnitude on analytical scans, and this engine already exists in Elasticsearch: it is the block-at-a-time machinery underneath ES|QL. Try both verbs:
Skip and sweep compose multiplicatively with everything before them: the query reads 3 of 50 columns, each column arrives 5 to 30 times smaller, most of its blocks are pruned unread, and what remains is swept at SIMD speed. No single step is magic. The stack of them is why the same question costs so much less against columns.
All of this raises an uncomfortable question. If columns are this good, and the columns already contain every field, what exactly is the stored document still for? Columnar mode’s answer is radical: nothing. The document stops being a thing that is stored and becomes a thing that is computed: to rebuild record #4812, read row 4812 of every column and reassemble the JSON. The document turns into an output format, a view over the columns, the way a rendered web page is a view over a database.
And the engine is lazy about it, which is the part that matters. Reconstruction is deferred until something actually needs the record, and for the analytical queries that dominate log workloads, nothing ever does: an aggregation sweeps columns and returns numbers, so the answer arrives and zero documents were ever assembled. The most common fate of a log document in columnar mode is to never exist at all. Watch both fates:
There is a name for the general principle: lazy materialization, and columnar engines live by it. Filter on cheap columns first, narrow the candidate rows, and only reconstruct wide values for the survivors, ideally for none. The document store spent fifteen years making the document the atom of everything; columnar mode demotes it to a query result.
Now the pieces assemble into the actual product. In columnar mode, the data exists once, in doc values. The stored _source is gone: regenerated on demand as in the last figure, and at petabyte scale some operators drop even the regeneration niceties and keep nothing but the columns. Inverted indexes stop being a default and become a choice you make per field: message keeps its full-text index because you genuinely grep it at 3 a.m.; trace_id might earn a keyword index if you chase traces; the forty fields you only ever aggregate get no index at all, because columns alone answer counting questions. The data model turns flat key-value, which for log data is less a restriction than an honest description. And the whole arrangement ships as a profile: columnar logs today, with vector and metrics profiles pointed at the same foundation. Every choice below has a price and a payoff; do the arithmetic yourself:
Read the verdicts closely, because they contain the honest fine print. Dropping an index never makes a question impossible, it makes it a column scan: correct, pruned by block metadata, swept vectorized, but linear where postings were logarithmic. Columnar mode does not abolish the trade-off between write-time work and read-time work; it hands you the dial, field by field, and defaults it to the questions log workloads actually ask.
Step far enough back and this essay is about fifteen years of industry divergence ending. One lineage perfected the document: search engines that could tokenize, rank, highlight and return rich records in milliseconds. Another perfected the column: analytical databases that could compress, prune and sweep trillions of values. Two literatures, two conference tracks, two sets of instincts, each quietly assuming the other’s workload was somebody else’s problem. Observability is the workload that refused to choose: it greps like search and counts like analytics, at volumes that punish any wasted byte. Columnar mode is the two lineages converging in one engine, behind one API.
The convergence is honest about its targets. Logs and observability, security telemetry, metrics and traces, operational analytics: data that is written once, read analytically, and retained long. It is equally honest about the non-targets. Update-heavy transactional flows stay in document modes, because mutating a record whose fields are scattered across compressed columns is exactly the operation this layout sacrifices. So do systems where document structure is user-facing semantics, product catalogs, content stores, anywhere nesting and shape are the meaning rather than transport packaging for flat facts.
And the migration story is deliberately boring, which for storage engines is the highest compliment: the same APIs and query languages work against both layouts, you migrate index by index, and document modes remain first-class rather than deprecated. Nothing about your queries has to know which physics runs underneath. That is the same quiet promise the metrics engine made when it went columnar first and proved that a search engine could store a signal once; the deeper background on the two structures this essay kept weighing against each other is in the doc values essay and the inverted index essay. The full announcement, benchmarks and roadmap are in the original article. Fifteen years of storing everything three ways, ending the way good engineering usually ends: by paying, at last, only for the questions you ask.