A log line has a lifetime. It is born messy in some process's stdout, gets parsed and enriched in flight, lands in an index, is queried constantly for a day, occasionally for a month, and then must be kept, cheaply, because the law says so. This essay follows that whole life.
Let's zoom out from structures to a lifetime. A log line's story begins before indexing, in an ingest pipeline: a chain of processors that parse, enrich and reshape it in flight. Try the three raw lines:
The indexed line lands in a data stream: an append-only front door backed by a rolling series of indices; when the current one grows big or old enough, it rolls over and a fresh one takes the writes. And then time does its work. Yesterday's logs are queried constantly; last quarter's, almost never, yet the law says keep them. So index lifecycle management migrates aging indices across hardware tiers, from fast expensive disks to object storage that costs almost nothing, while everything stays searchable. Drag time:
What happens to the log once it lands is the subject of Anatomy of a search; what happens when it is a metric instead is The metrics engine.