Eight layers sit between the app you type into and the GPU that does the math. Each one only works because the layer below it exists.
One prompt’s trip down the eight layers and back up. Click or tap a title or picture in the sketch to jump to its section.
1. Application
This is what people actually touch: a chat window, a coding assistant, a support bot. Most apps do no AI math themselves. They are ordinary software that wraps your request in instructions and context, then sends it to a model over an API.
When a task needs more than one question and one answer, a layer in between decides which tools to call, fetches documents, remembers earlier steps, and loops until the job is done. Protocols like MCP, introduced in November 2024, let a model plug into tools and data in a standard way. One agent task can mean dozens of model calls, so this layer multiplies the compute behind every request.
A model is a very large file of numbers, called weights, plus the architecture that says how to combine them. Text is cut into tokens, and the model predicts the next one, over and over. Its parameter count decides how much memory it needs just to exist, before it does any work.
Serving software turns one model into something thousands of people can use at once. Each request has two phases. Prefill reads your whole prompt in parallel, which is heavy on math. Decode then writes the answer one token at a time, re-reading the model’s weights and a running notebook of the conversation, the KV cache, for every token. Serving software mostly batches requests and manages that cache so the hardware stays busy.
PyTorch and JAX turn “multiply these matrices” into instructions a chip can run. For Nvidia, that path runs through CUDA and its libraries, which developers have written against for nearly two decades. That software, and the habit around it, is a large part of Nvidia’s moat. Google’s TPUs use their own compiler stack instead.
Memory sets the speed limit. HBM, high-bandwidth memory, is stacked right beside the GPU and holds the weights and the KV cache. Larger, slower DRAM attached to the CPU holds the overflow, and SSDs hold the files at rest. Decode is limited by how fast bytes reach the compute, not by how fast the compute is. Nvidia’s Rubin supports up to 288 GB of HBM4 and AMD’s MI455 up to 432 GB, and Micron has said its entire 2026 HBM4 capacity is sold out.
SK hynix (000660.KS) — makeSamsung (005930.KS) — makeMicron (MU) — make
7. CPU
The CPU is the manager. It runs the operating system, handles networking and storage, schedules work, and feeds data to the GPUs. It does very little of the model math. Designs like Nvidia’s Grace pair the CPU tightly with the GPU so the two can share memory quickly: a GB200 NVL72 rack holds 36 Grace CPUs alongside 72 Blackwell GPUs.
The GPU does the matrix multiplication, thousands of operations at once. Everything above it exists to keep it fed, and everything below it, from power to cooling to the building, exists to keep it running. Custom accelerators such as Google’s TPU do the same job with a narrower design.
Training reads and writes enormous amounts of data and is limited mostly by compute and by the network between GPUs. Inference is limited mostly by memory, because decode re-reads the weights for every token. That is why HBM, layer 6, has become some of the most contested silicon in the stack.