If you have read the essays on this site, you already know more about large language models than you think. You know what a vector embedding is and why nearby points mean similar things. You know how HNSW races through a billion of them. You know what a retrieval system does when a query arrives: score every candidate against it, keep the winners. A transformer is built from exactly these parts, rearranged and made differentiable. This essay walks the whole pipeline with figures you can drag; for the same journey in careful linear prose, read 0xkato's excellent How LLMs actually work, the companion piece this one turns into machinery.
The pipeline has seven stations: text becomes tokens, tokens become vectors, vectors get position stamped in, attention lets them retrieve from each other through many parallel heads, a feed-forward network transforms what was retrieved, a residual stream carries everything through the depth, and a sampler turns the final scores into one emitted token. Then the loop repeats, leaning on a handful of engineering tricks that keep it affordable. Every station gets a figure.
The model never sees text. Before anything else runs, a tokenizer chops the input into pieces from a fixed vocabulary and replaces each piece with an integer. The vocabulary is learned once, before training, by greedily merging the character pairs that co-occur most often (the family of algorithms called byte-pair encoding), so common words survive as single pieces while rare words shatter into fragments. Subwords are the working compromise: whole-word vocabularies are enormous and helpless before tomorrow's words, character vocabularies are tiny but stretch every sentence into thousands of steps, and pieces sit in between. Type into the box and watch the chopping:
strawberry: it became two pieces, two integers, and the letters are gone. Ask a model how many r's the word contains and it must answer from IDs that never encoded an r count. The famous failure is not stupidity; it is the input format.Two things matter about the integer. First, it is nothing but a row index: token 48021 means "row 48021 of the embedding table", the way a doc ID in Lucene means a slot in a column, and carries no meaning by itself. Second, the vocabulary is finite and surprisingly small: tens of thousands of entries in older models, a few hundred thousand in recent ones, covering every language, every programming language, and every typo humanity produces, because anything not in the table can always fall back to smaller fragments, down to single bytes if necessary. Nothing is ever out of vocabulary; some things are just expensive to spell. The chopping recipe itself varies by family: the GPT line uses byte-level byte-pair encoding, LLaMA 1 and 2 used SentencePiece before LLaMA 3 moved to a GPT-style BPE vocabulary, and the difference shows up as differently shaped chips for the same text.
Row lookup turns each token into a vector, a few thousand numbers (4,096 per token in a 7B-class model, wider as models grow), and here you are on home ground: this is the same geometry as the vector database essay, where similar meanings sit at small cosine distances. The astonishing part is where the geometry comes from. Nobody labels it. The embedding table starts as noise and is dragged into shape by one pressure only, predicting the next token, and out of that pressure falls structure nobody programmed: directions that mean gender, tense, capital-of. Try the arithmetic:
Hold on to this picture, because everything downstream is operations on it. A transformer layer does not manipulate symbols; it moves points around this space, nudging the vector for "bank" toward the river cluster or the finance cluster depending on neighbors. The vector that finally exits the last layer is not a word; it is a position in meaning-space, and the model's last act will be asking which vocabulary rows sit closest to it.
Attention, coming next, is a bag operation: it scores pairs of vectors and has no idea which came first. But "dog bites man" and "man bites dog" had better not collapse into the same bag, so position must be written into the vectors themselves. The trick modern models use, rotary position embedding (RoPE), is almost embarrassingly elegant: take each 2D slice of the query and key vectors and rotate it by an angle proportional to the token's position. One line of trig is all you need: the angle is theta = position × frequency.
The invariance is the whole point. Because both vectors rotate, the dot product between a query at position p and a key at position q depends only on the gap p − q, never on the absolute positions. "Three tokens back" means the same thing at position 40 as at position 40,000. Relative is not the same as unlimited, though: the slow hands make rotations a model trained on short documents has never seen, so stretching a model past its training length still needs extra work (rescaling the frequencies, as position interpolation and YaRN do, plus some training at the new length). Dozens of slices at different frequencies run at once, like the hands of a clock: fast hands distinguish adjacent tokens, slow hands measure paragraphs.
It was not always done this way. The original 2017 transformer added sinusoidal position signals directly into the embeddings, which worked, but forced one vector to carry two jobs at once, meaning and address. Rotation (published by Su et al. in 2021 and now the default in LLaMA, Mistral, Gemma, Qwen and most open-weight families) moves position out of the embedding and into the query-key comparison itself, and it adds no parameters at all: the angles are computed, never learned. One honest caveat survives every scheme so far: models use the start and the end of a long prompt more reliably than the middle, the "lost in the middle" effect measured by Liu et al. in 2023, which is why the folk advice to put critical context first, or repeat it last, actually works.
Now the station the architecture is named for, and the one a search engineer already owns. Every token derives three vectors through learned projections: a Query, "what am I looking for"; a Key, "what I can be found by"; a Value, "what I hand over on a match". Each token's query is scored against every other token's key by dot product, the scores are scaled and pushed through a softmax, and the token collects a weighted blend of the values. Read that again as a search person: a query, matched against index keys, returning stored documents. It is the anatomy of a search with one word changed: instead of a hard WHERE clause that keeps some documents and drops the rest, softmax is a soft WHERE clause that keeps everything in proportion to its match score. Retrieval, made differentiable so it can be learned end to end.
was and watch cat dominate: the verb needs its subject, six tokens away, and attention fetches it in one hop with no notion of distance. The figure ignores the causal mask for clarity; in a real decoder each token can only attend backward.Two details complete the picture. First, the causal mask: a generating model reads left to right, so a token at position 5 may retrieve from positions 1 through 5 and nothing later; it cannot see tokens that do not exist yet. The implementation is blunt: future positions get their match scores forced so low that softmax hands them effectively zero weight. The search analogy holds even here: it is a filter applied to the index before scoring. Second, some heads implement retrieval so literally that researchers could name them. Induction heads, described by Anthropic in 2022, spot the pattern "A B ... A" and predict B: they look back to where A last appeared, see what followed, and copy it. An exact-match lookup, learned from scratch, and one of the clearest known mechanisms behind in-context learning: the model reuses a phrase from your prompt because a head literally goes and fetches it.
The price is written in the arithmetic: every token scores against every token it is allowed to see, so doubling the prompt roughly quadruples the work. Hold that thought; it is the entire reason the making-it-fast machinery at the end of this essay exists.
One retrieval pass cannot serve every need at once. A sentence has a subject and a verb agreeing across a gap, pronouns pointing back at names, phrases nesting locally, references arcing between sentences, and a single set of Q, K, V projections would have to score all of it with one ranking function. So attention runs many times in parallel, and here is the detail almost everyone gets wrong: a head is not handed a slice of the token vector. Each head has its own learned projection matrices that map the full vector, all 4,096 numbers of it, down into its own small working space: 128 dimensions per head when 32 heads share a 4,096-wide model. Thirty-two complete, independent views of the whole token, not thirty-two fragments of it.
The head outputs are concatenated and pushed through one final linear layer, which learns how to blend the parallel retrievals back into a single vector. Nobody assigns the specialties: the model is never told "head 7 shall track grammar", yet inspection after training keeps finding heads that follow subject-verb structure, heads that resolve which name a pronoun means, heads locked to fixed positional offsets, and the induction heads from the previous section. Multiply it out, 32 heads per layer across dozens of layers, and a frontier model is running thousands of specialized search engines over your prompt at once, each with its own learned ranking function, all querying the same corpus: the sentence so far.
Attention moves information between tokens; it does not do much to it. That job belongs to the other half of every layer, the feed-forward network, which runs on each token's vector independently, no cross-token mixing at all, and it is where most of the model actually lives: roughly two thirds of the parameters sit here, not in attention. The shape is always the same: expand the vector to about four times its width, apply a nonlinearity that lets some dimensions through and kills others, compress back down.
The middle step, the gate, is the entire reason depth means anything. Pure linear math collapses: two matrix multiplications in a row are mathematically one matrix multiplication, so a hundred purely linear layers have exactly the power of a single one. The nonlinearity breaks the collapse, and its shape has been refined across generations, ReLU in the original transformer, GELU in GPT and BERT, SwiGLU in LLaMA, Mistral and PaLM. SwiGLU is a literal gate, one projection deciding how much of another passes through, which is what the threshold slider above caricatures; it is also why modern expansion sizes drift off the clean 4x. A gated layer carries a third weight matrix, so designers shrink the hidden width to keep the parameter count level: LLaMA-2 7B uses 11,008, about two thirds of 4 × 4,096, while Mistral 7B spends more and uses the 14,336 quoted above.
The best current reading of this structure is key-value memory, one more retrieval system hiding in plain sight: the expansion weights act as keys that detect patterns in the incoming vector, the gate decides which detected patterns matter, and the compression weights hold the values written back. Interpretability work keeps finding hidden units whose firing patterns are legible, and the model-editing literature exploits the structure directly: ROME, rank-one model editing (Meng et al., 2022), changes "the Eiffel Tower is in Paris" to "in Rome" with one rank-one edit to a single mid-layer FFN weight matrix, and the model then generates text consistent with the association it never learned, because that is where the facts are. Honest caveat: most units are not so tidy, and most facts smear across many layers. But as a first approximation, attention decides where to look, and the FFN decides what it means.
Everything so far sounds like an assembly line: attention rebuilds the token's vector, the FFN rebuilds it again, thirty times over. That is not what happens, and the difference is the deepest design decision in the architecture. A sub-block's output does not replace the vector; it is added to it: new vector = old vector + block output. Every head and every FFN writes a small delta onto a shared running total, the residual stream, and the original embedding keeps a direct additive path all the way into the final layer. The idea arrived from image recognition (ResNet, He et al., 2015), where it solved a brutal practical problem: deep networks would not train, because the learning signal died before it reached the early layers. Give the signal a shortcut straight through and suddenly hundreds of layers train. Watch what each policy does to a vector crossing thirty-two layers:
The additions need policing. Numbers flowing through dozens of sums either explode upward or decay toward zero, so between sub-blocks a normalization step rescales each token's vector back into a controlled range. Two refinements became standard. Modern models normalize before each sub-block rather than after (pre-norm, GPT-2 onward), which is much of why very deep stacks train at all; and they use RMSNorm (LLaMA, Mistral, Gemma), which drops classic layer norm's mean-subtraction step and keeps only the rescale: most of the benefit, less arithmetic. Hold on to this picture for one more reason: interpretability research now treats the residual stream as the central object of the whole model, because every component, every head, every FFN, even the final unembedding, reads from it and writes back to it. If you want the storage analogy, it is the shared append-only log the entire cluster commits to.
After the last layer, generation cares about exactly one vector: the residual stream above the final token. It is multiplied against the vocabulary table one last time (the unembedding), producing one raw score for every token the model could emit next: the logits. They are not probabilities; they are unbounded, positive or negative, just how strongly the final vector points at each row. A softmax turns them into a proper distribution, and only then do the decoding dials apply. Temperature divides the logits before the softmax, sharpening or flattening the whole distribution; top-k keeps only the k highest candidates; top-p, its adaptive cousin and the one on the slider below, trims the list to the smallest set that covers probability p. The prompt is "The capital of France is":
This is the only place the design asks for randomness. Everything upstream, embeddings, rotations, attention, FFN, is arithmetic meant to give the same answer every run (in practice, batched floating-point math on GPUs can add tiny run-to-run differences); the dice are thrown once per token, right here. Greedy decoding (T → 0) sounds like it should be best, and for open-ended writing it is not: it tends to produce repetitive, self-trapping text, so practical systems sample from a sculpted distribution instead. And note what the model emits: one token. A paragraph is hundreds of trips around this loop, each new token appended to the input for the next trip: new attention, new FFN, new final vector, new dice, until an end-of-sequence token comes up or a length limit calls time. Run naively, every trip would reprocess the entire growing prefix from scratch, which brings us to the machinery that makes the loop affordable.
The loop as described is ruinous: attention already scores every token against every earlier token, and redoing all of it for the whole prefix on every new token would bury any long conversation. The fix rests on one observation: the past does not change. Once a token's Key and Value vectors are computed they are final, so the model stores them: the KV cache. Each new token computes only its own Q, K and V, scores its single query against every cached key, appends its own K and V to the cache, and moves on. Without the cache, producing token N means reprocessing all N − 1 predecessors; with it, token N costs one row of attention. Drag the slider and watch the curves separate:
The cache trades compute for memory, and at long context the memory wins the argument: the KV cache is often the dominant memory cost of serving an LLM, which is what limits how many conversations fit on one GPU. Hence grouped-query attention: keep all the query heads, but let groups of them share a much smaller set of Key/Value heads. LLaMA-2 70B runs 64 query heads against 8 K/V heads; Mistral 7B runs 32 against 8. Accuracy stays close to full multi-head attention while the cache shrinks by the same factor as the heads, exactly what the slider above shows.
Two more tricks round out the modern serving stack. Mixture of experts attacks the FFN, where most of the parameters live: split it into many parallel FFNs, the experts, and train a tiny router that picks a few of them per token. Mixtral 8x7B carries 8 experts per layer and activates 2 for any given token: 46.7 billion parameters on disk, roughly 12.9 billion doing work per token. Parameter count and per-token compute, which used to be the same dial, decouple: the model can know more without every token paying for all of it.
And speculative decoding attacks the loop itself. Generation is serial, one token per trip, but verification is parallel: a transformer can score every position of a candidate sequence in a single pass. So a small, fast draft model proposes several tokens ahead, the big model checks them all at once, keeps the longest prefix consistent with its own probabilities, and corrects the first disagreement. Done correctly, the output distribution is mathematically identical to running the big model alone; the speedup is pure profit whenever the draft guesses well, which for predictable stretches of text is often.
Nothing above is secret. The open-weight families publish their architectures, and as far as anyone outside the closed labs can tell, GPT, Claude and Gemini are built on the same skeleton: the tokenize-embed-rotate-attend-transform-sample pipeline, published in 2017 and refined in public since. Even the refinements travel together: pre-norm placement, RMSNorm, RoPE, SwiGLU, grouped-query attention, experts at the top end, five years of accumulated consensus layered on the original design. What separates the models is everything the skeleton does not fix: the configuration (layer count, vocabulary size, head counts, dense FFN or experts), what data flowed through during training and at what scale, and what happened afterward. Because the pretrained object, the base model, is trained on next-token prediction alone. Not facts, not conversation, not helpfulness, not reasoning: one loss, guess the next token, trillions of times. Ask a base model a question and it is as likely to continue with three more questions as to answer, because both continuations are common on the internet. The assistant character is manufactured in post-training: supervised fine-tuning on demonstrations, then reinforcement learning against preference and correctness signals, and the recipes there differ far more between labs than the architectures do.
So the whole machine in one sentence: text is chopped into tokens, tokens index into learned vectors, rotation stamps position into them, attention lets every token run a soft retrieval over its past through thousands of parallel heads, the FFN transforms what came back through the layers where the facts live, every block adds its delta to a residual stream that normalization keeps sane, the final vector is scored against the vocabulary, and a temperature-shaped throw of the dice picks one token, whereupon the loop begins again with the KV cache from the previous section carrying the past. Architectures will keep shifting under our feet; state-space models like Mamba are already at the door, and hybrids are being tried. But sequence models cannot dodge the problems themselves: representing meaning, encoding order, moving information across distance, storing what was learned, staying numerically alive through depth, choosing what to say next. Whatever machinery wins, these are the stations it will have to build.