What happens between hitting Enter and watching the answer stream in. The model stays a black box here: this is about who does what, and in what order.
1. The model is the recipe, the GPU is the cook
The picture I keep coming back to is a kitchen. The model is a recipe: a very large file of numbers, called weights, that says how to turn the text so far into the next word. A recipe on its own cooks nothing. Something has to follow it.
One question, one server, one next word, then round again. Click or tap any part of the sketch to jump to it.
Your question. It leaves your app as text and lands on a server in a data center. (The cloud, it turns out, is mostly a building.)
Serving software. This is the front desk. It accepts requests from lots of people at once, puts them in line and decides what runs next. It runs on the CPU.
The CPU. The manager. It does very little of the model math itself. It gets the work ready and hands it to the GPU.
The GPU. The cook. It does the math, thousands of operations at once, and what comes out is one next word. That word is added to the text, and it goes round again until the answer is done. It keeps notes on the words it has already read, so each round doesn’t start from scratch.
The recipe. The model’s weights are loaded into the GPU’s own memory before any question shows up, and they stay there. The cook keeps the recipe on the counter, not in a filing cabinet down the hall.
The circled numbers. In the AI stack, the model is layer 3, serving is 4, memory is 6, the CPU is 7 and the GPU is 8. Your question does start at layer 1, the app. But after that, follow it through the sketch and you get 4, 7, 8, with 3 sitting inside 6. So the layer numbers are not the route your question takes. They say how close each piece sits to the hardware: the bigger the number, the closer to the metal.
2. Every request: prefill on the question, decode on the answer
Every request gets the same two steps, and they behave very differently.
One request: a long read, then a run of small steps. Click or tap a part to jump to it.
Prefill. The model reads your question first. Every word of it is already known, so the GPU can push all of them through the model at the same time, in one big parallel pass. That is a lot of math at once, and it is most of the pause before anything appears on screen. A longer question means a longer pause.
Decode. Then it writes the answer, one token at a time (a token is a word or a piece of a word). It can’t skip ahead: each new word depends on the words before it, and the next word doesn’t exist until this one does. So the answer comes out as a row of small steps, one per token. That is the streaming you watch.
So the question gets read in one go, and the answer gets written by a very careful typist who refuses to guess ahead. The two steps also strain the GPU in different ways: prefill keeps its math busy, while each decode step does little math and is held back more by how fast the GPU can read its memory.
Engineers have short names for the two waits you feel: TTFT (time to first token) is the wait before the first word, and TPOT (time per output token) is the gap between words.
3. Sharing one GPU: the shuttle bus
One GPU doesn’t serve one person at a time. The picture that made it click for me is a shuttle bus. Every decode step is one bus trip, and each trip carries one new word for every rider. A trip with 50 riders takes longer than a trip with 2, but nowhere near 25 times longer, so each rider’s share of the trip gets much cheaper. Engineers call filling the bus batching.
Each bus trip carries one new word for every rider. How full you let the bus get is the batch size.
The catch: a fuller bus takes longer per trip, so every rider’s next word arrives a bit later (TPOT goes up). And once the bus is completely full, new people wait at the stop before they can even board (TTFT jumps). So how full you let the bus get is a dial between cost and speed. A chat app keeps seats free so words stream quickly; an overnight job that summarizes a million documents packs the bus, because nobody is watching the words appear.
My first instinct was to change the dial with the time of day. The twist: at busy times the buses fill up on their own, so to stay fast you add more buses (more GPUs) rather than emptying each one. At quiet times the seats are free anyway, so you fill them with work nobody is waiting on. Same instinct, different levers.
The one-liner. Batching raises throughput and lowers the cost per token, but adds latency for each user, so you size batches to the product’s speed target.