AI, As I Go

Inference & serving

How your question becomes an answer

What happens between hitting Enter and watching the answer stream in. The model stays a black box here: this is about who does what, and in what order.

1. The model is the recipe, the GPU is the cook

The picture I keep coming back to is a kitchen. The model is a recipe: a very large file of numbers, called weights, that says how to turn the text so far into the next word. A recipe on its own cooks nothing. Something has to follow it.

Your question, one server, one next word A hand-drawn sketch. Your question goes into one server. Inside it, serving software takes the request and puts it in line, and the CPU, the manager, hands the work to the GPU. The GPU is the cook and does the math. In the GPU's own memory sits the model's weights, the recipe. Out comes the next word, which is added to the text, and the loop goes round again until the answer is done. Small circled numbers show each part's layer number from the AI stack page: serving 4, CPU 7, GPU 8, memory 6, model 3. one server, in a data center added to the text, then round again until done Read about your question ↓ your question “why is the sky blue?” Read about serving software ↓ serving software takes your request, puts it in line 4 Read about the CPU ↓ CPU the manager: hands the work to the GPU 7 Read about the GPU ↓ GPU = the cook does the math 8 Read about the recipe in the GPU's memory ↓ its own memory model weights = the recipe 6 3 Read about the next word ↓ next word Read what the circled numbers mean ↓ circled numbers? layers from the stack page. Your question goes 4 → 7 → 8, and 3 sits inside 6. Not a route: a distance from the hardware. Your question, one server, one next word A hand-drawn sketch, top to bottom. Your question goes into one server. Inside it, serving software takes the request and puts it in line, and the CPU, the manager, hands the work to the GPU. The GPU is the cook and does the math. In the GPU's own memory sits the model's weights, the recipe. Out comes the next word, which is added to the text, and the loop goes round again until the answer is done. Small circled numbers show each part's layer number from the AI stack page: serving 4, CPU 7, GPU 8, memory 6, model 3. one server, in a data center added to the text, then round again until the answer is done Read about your question ↓ your question “why is the sky blue?” Read about serving software ↓ serving software takes your request, puts it in line 4 Read about the CPU ↓ CPU the manager: hands work to the GPU 7 Read about the GPU ↓ GPU = the cook does the math 8 Read about the recipe in the GPU's memory ↓ its own memory model weights = the recipe 6 3 Read about the next word ↓ next word Read what the circled numbers mean ↓ circled numbers? layers from the stack page. Your question goes 4 → 7 → 8, and 3 sits inside 6. Not a route: a distance from the hardware.
One question, one server, one next word, then round again. Click or tap any part of the sketch to jump to it.

Your question. It leaves your app as text and lands on a server in a data center. (The cloud, it turns out, is mostly a building.)

Serving software. This is the front desk. It accepts requests from lots of people at once, puts them in line and decides what runs next. It runs on the CPU.

The CPU. The manager. It does very little of the model math itself. It gets the work ready and hands it to the GPU.

The GPU. The cook. It does the math, thousands of operations at once, and what comes out is one next word. That word is added to the text, and it goes round again until the answer is done. It keeps notes on the words it has already read, so each round doesn’t start from scratch.

The recipe. The model’s weights are loaded into the GPU’s own memory before any question shows up, and they stay there. The cook keeps the recipe on the counter, not in a filing cabinet down the hall.

The circled numbers. In the AI stack, the model is layer 3, serving is 4, memory is 6, the CPU is 7 and the GPU is 8. Your question does start at layer 1, the app. But after that, follow it through the sketch and you get 4, 7, 8, with 3 sitting inside 6. So the layer numbers are not the route your question takes. They say how close each piece sits to the hardware: the bigger the number, the closer to the metal.

2. Every request: prefill on the question, decode on the answer

Every request gets the same two steps, and they behave very differently.

One request on a timeline: prefill, then decode A hand-drawn timeline. You hit Enter. Prefill reads the whole question at once: the words why, is, the, sky, blue go in together, because every question word is already known. The length of that block is TTFT, the wait for the first word. Then the first word shows up and decode writes the answer one word at a time: Sunlight, scatters, off, air, each needing the word before it. The gap between two words is TPOT. Then the answer is done. time you hit Enter first word shows up! Read about TTFT ↓ TTFT = the wait Read about prefill ↓ PREFILL reads the whole question at once why is the sky blue? all together, in one pass why? every question word is already known Read about decode ↓ DECODE writes the answer one word at a time Sunlight scatters off air… done why? each word needs the one before it Read about TPOT ↓ TPOT gap per word One request on a timeline: prefill, then decode A hand-drawn timeline, top to bottom. You hit Enter. Prefill reads the whole question at once: the words why, is, the, sky, blue go in together, because every question word is already known. The length of that block is TTFT, the wait for the first word. Then the first word shows up and decode writes the answer one word at a time: Sunlight, scatters, off, air, each needing the word before it. The gap between two words is TPOT. Then the answer is done. you hit Enter time first word shows up! Read about prefill ↓ PREFILL reads the whole question at once why is the sky blue? all together, in one pass why? every question word is already known Read about TTFT ↓ ↑ TTFT = this whole wait Read about decode ↓ DECODE writes the answer one word at a time Sunlight scatters off air… needs the last word why? each word needs the one before it Read about TPOT ↓ TPOT per word done
One request: a long read, then a run of small steps. Click or tap a part to jump to it.

Prefill. The model reads your question first. Every word of it is already known, so the GPU can push all of them through the model at the same time, in one big parallel pass. That is a lot of math at once, and it is most of the pause before anything appears on screen. A longer question means a longer pause.

Decode. Then it writes the answer, one token at a time (a token is a word or a piece of a word). It can’t skip ahead: each new word depends on the words before it, and the next word doesn’t exist until this one does. So the answer comes out as a row of small steps, one per token. That is the streaming you watch.

So the question gets read in one go, and the answer gets written by a very careful typist who refuses to guess ahead. The two steps also strain the GPU in different ways: prefill keeps its math busy, while each decode step does little math and is held back more by how fast the GPU can read its memory.

Engineers have short names for the two waits you feel: TTFT (time to first token) is the wait before the first word, and TPOT (time per output token) is the gap between words.

3. Sharing one GPU: the shuttle bus

One GPU doesn’t serve one person at a time. The picture that made it click for me is a shuttle bus. Every decode step is one bus trip, and each trip carries one new word for every rider. A trip with 50 riders takes longer than a trip with 2, but nowhere near 25 times longer, so each rider’s share of the trip gets much cheaper. Engineers call filling the bus batching.

Batching as a shuttle bus Three hand-drawn buses. Few riders: fast words but expensive per person. Half full: the sweet spot. Full: cheap per person but each word comes slower, and new people wait at the stop, so the wait for the first word jumps. few riders fast words pricey per person half full the sweet spot full cheap per person slower words (TPOT up) ← waitingTTFT jumps one bus trip = one decode step
Each bus trip carries one new word for every rider. How full you let the bus get is the batch size.

The catch: a fuller bus takes longer per trip, so every rider’s next word arrives a bit later (TPOT goes up). And once the bus is completely full, new people wait at the stop before they can even board (TTFT jumps). So how full you let the bus get is a dial between cost and speed. A chat app keeps seats free so words stream quickly; an overnight job that summarizes a million documents packs the bus, because nobody is watching the words appear.

My first instinct was to change the dial with the time of day. The twist: at busy times the buses fill up on their own, so to stay fast you add more buses (more GPUs) rather than emptying each one. At quiet times the seats are free anyway, so you fill them with work nobody is waiting on. Same instinct, different levers.

The one-liner. Batching raises throughput and lowers the cost per token, but adds latency for each user, so you size batches to the product’s speed target.