Growing old cheaply

The Elasticsearch essays · May 2026

Yesterday's logs are queried constantly; last quarter's, almost never, yet the law says keep them. The economics of observability are decided by what happens to data after its first week. This essay follows an index from its busy youth to its very cheap old age.

A clean document needs an address, and for logs the address is a data stream: one stable name to write to, backed by a rolling series of hidden indices. When the current backing index gets too big or too old (the built-in logs policy says 50 GB per primary shard or 30 days), the stream rolls over: a fresh index silently takes the writes and the old one is sealed, never to be written again, which is exactly what the lifecycle machinery below needs. Run a month of logging:

logs-payments · one data stream, 30 days of writes

backing indices

–

what triggers the roll

ingest rate8 GB/day

max shard size50 GB

max age7 d

Each chip is one backing index; its width is how many days it stayed the write target. Crank the ingest rate: size starts triggering rollovers before age does, and the indices multiply. The reader never sees any of this; queries address the stream, which fans out to every backing index behind it.

And then time does its work. Remember the opening asymmetry: the newest data gets almost every query, the oldest almost none. So index lifecycle management migrates aging indices across hardware tiers, from fast expensive disks to object storage that costs almost nothing, while everything stays searchable. Drag time:

Hot

fast SSD · takes all writes

Warm

cheaper disk · read-only

Cold

no replicas · restores from snapshot

Frozen

object storage · searched directly

actual cost

if everything stayed hot

months of retention6

Each chip is one month of logs. Policy: hot for 1 month, warm until 3, cold until 9, frozen after. The cost per tier is relative (a frozen month costs about a twentieth of a hot one), but the shape is real: drag retention out to 18 months and the two bars diverge, because every month you add lands in the cheapest tier. The frozen tier is why keeping years of logs stopped being a luxury.

The frozen tier deserves a sentence of awe: those indices live as searchable snapshots in object storage, and are queried directly there, with local caching of just the blocks a query touches. Sealed, immutable indices (thank segments) are what make that safe. And the newest chapter of this story is storing the log data columnar from birth: Columnar logs, stored once.

Shrinking: fewer shards for quieter data

An index sharded six ways made sense when it was ingesting a firehose; a week later the same index is read-only and those six shards are just overhead: each one is a Lucene index with its own segments, its own memory footprint, its own entry in every cluster-state update. The warm phase can shrink it:

–

per-shard overhead · before

–

after shrink

shrink to

Shrinking to 1 shard turns six sets of everything into one; the target must divide the source count evenly, so six shards can become three, two, or one. The mechanics are pure segment reuse: once a copy of every shard sits on one node, the new index hard-links their files instead of copying them, so shrinking 200 GB takes seconds, immutability's dividend yet again. Multiply by the hundreds of indices a logging cluster rolls over every week, and shrink is the difference between ten thousand shards and two thousand: the master node notices.

The frozen tier: searching what isn't there

The coldest data doesn't live on the node at all: it lives in object storage, and the node keeps only a cache of the pieces recent queries actually touched. Whether that feels like local disk or like the cloud depends on one ratio:

–

typical query latency

–

data per node

local cache size

how repetitive are your queries?

A node fronting 5 TB of snapshotted indices with a cache you size. Reads are skewed: an investigation hammers the same incident window again and again, so a cache holding a fraction of a percent of the data can serve most reads. Slide the queries toward random scans (an auditor reading across years) and the same cache stops helping; every miss is an object-store fetch at tens of milliseconds per block. The numbers come from a Zipf model of access, the shape real query traffic tends to have. The frozen tier's bet is that old data is queried rarely and narrowly, and for logs that bet almost always pays.