Ingest: from chaos to schema

The Elasticsearch essays · April 2026

Every log line is born messy: a process prints whatever its author felt like printing, in whatever shape. Between that chaos and a queryable document sits the ingest pipeline, a chain of processors that parse, enrich and reshape data in flight. This essay is about that chain, and about the pattern language at its heart.

Here is the whole chain at work on three lines from three different programs. Pick one and follow it down:

1 · grok: pattern-match fields out of the text (every capture is a string)

2 · enrich: look up what the data can't say itself

3 · normalize: canonical names and real types

4 · the document that gets indexed

Three very different raw lines, one pipeline. Grok cuts the text into named strings; enrich looks up what the line cannot say about itself (which service, which team, which region); normalization renames everything to the common schema and turns "500" into the number 500, so a range query on status codes works. Amber chips are fields that did not exist in the raw text; green ones show each rename (and every string that became a number).

Grok: patterns you can hold

The workhorse processor is grok: regular expressions wearing name tags. A pattern like %{IP:client} says "an IP address goes here, call it client", and a full pattern is just those pieces interleaved with the literal text you expect between them. The only way to really understand grok is to break one, so this pattern is editable:

Three raw lines against your pattern. Green chips are captured fields; when a line fails, its text turns orange from the exact point where the pattern gave up. Try: delete the ms literal at the end and watch line 3 recover (and lines 1 and 2 break); change %{NUMBER:status:int} to %{WORD:status} and the number becomes a string again. Supported here: IP, WORD, NUMBER, NOTSPACE, GREEDYDATA, with an optional :int or :float to type the capture.

After extraction comes normalization: the Elastic Common Schema and OpenTelemetry's semantic conventions give the fields canonical names, so client becomes source.ip in every index, from every source, and one dashboard fits all of them. Parsing gives you fields; a schema is what makes them the same fields everywhere.

When parsing fails

Every pipeline meets a line it cannot parse: a truncated message, an app that changed its log format at 3 a.m., a stack trace where a status code should be. What happens next is not fate, it is policy: you choose it per pipeline with on_failure, or per data stream with its failure store:

Notice what the good policies share: none of them pretends the line was parsed, and none of them throws it away. A pipeline is a promise about structure, and the honest version of that promise includes the lines it could not keep.

Once clean, the line needs somewhere to live and a plan for growing old; both are the subject of Growing old cheaply. What queries can do with it afterwards is Anatomy of a search.