AI, As I Go

Infrastructure

The AI stack: from app to GPU

Eight layers sit between the app you type into and the GPU that does the math. Each one only works because the layer below it exists.

What happens when you hit Enter A hand-drawn trip of one prompt through the eight layers of the AI stack. You hit Enter, and the prompt goes down: 1 app wraps your prompt in instructions and context and sends it over an API; 2 agents plan steps and call tools, so one task can mean dozens of model calls; 3 the model cuts text into tokens and predicts the next one, over and over; 4 serving batches you with other users and keeps the KV cache; 5 frameworks like PyTorch and CUDA turn matrix math into chip instructions; 6 memory: HBM holds the weights and KV cache right beside the GPU; 7 the CPU, the manager, schedules work and feeds the GPUs; 8 the GPU does the matrix math, thousands of operations at once. The answer then comes back up the stack. Callout: memory sets the speed limit, because decode re-reads the weights for every token. you hitEnter yourpromptgoesdown answer comes back up(one token at a time) software hardware Read about layer 1: application ↓1 app wraps your prompt in instructions+ context, sends it over an API Read about layer 2: orchestration and agents ↓2 agents plans steps, calls tools — one taskcan mean dozens of model calls Read about layer 3: the model ↓3 model cuts text into tokens, predictsthe next one, over and over Read about layer 4: inference serving ↓4 serving batches you with other users,keeps the KV cache Read about layer 5: frameworks and drivers ↓5 frameworks PyTorch / CUDA turn matrix mathinto chip instructions Read about layer 6: memory ↓6 memory HBM Read about layer 7: cpu ↓7 CPU the manager: schedules work,feeds the GPUs Read about layer 8: gpu ↓8 GPU busy! does the matrix math,thousands of ops at once Read where the bottleneck sits ↓memory sets the speed limit HBM holds the weights + KV cache why? decode re-reads the weights for every token, so bytes reachingthe GPU — not raw compute — decide how fast the answer streams What happens when you hit Enter A hand-drawn trip of one prompt through the eight layers of the AI stack. You hit Enter, and the prompt goes down: 1 app wraps your prompt in instructions and context and sends it over an API; 2 agents plan steps and call tools, so one task can mean dozens of model calls; 3 the model cuts text into tokens and predicts the next one, over and over; 4 serving batches you with other users and keeps the KV cache; 5 frameworks like PyTorch and CUDA turn matrix math into chip instructions; 6 memory: HBM holds the weights and KV cache right beside the GPU; 7 the CPU, the manager, schedules work and feeds the GPUs; 8 the GPU does the matrix math, thousands of operations at once. The answer then comes back up the stack. Callout: memory sets the speed limit, because decode re-reads the weights for every token. you hitEnter prompt goes down the blue arrow,the answer comes back up the copper one Read about layer 1: application ↓1 app wraps your prompt, sends it to a model Read about layer 2: orchestration and agents ↓2 agents plans, calls tools — dozens of calls Read about layer 3: the model ↓3 model predicts the next token, again & again Read about layer 4: inference serving ↓4 serving batches users, keeps the KV cache Read about layer 5: frameworks and drivers ↓5 frameworks matrix math → chip instructions Read about layer 6: memory ↓6 memory HBM Read about layer 7: cpu ↓7 CPU the manager: feeds the GPUs Read about layer 8: gpu ↓8 GPU does the math, thousands of ops at once busy! Read where the bottleneck sits ↓memory sets the speed limit
One prompt’s trip down the eight layers and back up. Click or tap a title or picture in the sketch to jump to its section.

This is what people actually touch: a chat window, a coding assistant, a support bot. Most apps do no AI math themselves. They are ordinary software that wraps your request in instructions and context, then sends it to a model over an API.

Microsoft (MSFT) — software Salesforce (CRM) — software Adobe (ADBE) — software ServiceNow (NOW) — software

When a task needs more than one question and one answer, a layer in between decides which tools to call, fetches documents, remembers earlier steps, and loops until the job is done. Protocols like MCP, introduced in November 2024, let a model plug into tools and data in a standard way. One agent task can mean dozens of model calls, so this layer multiplies the compute behind every request.

Salesforce (CRM) — agents Microsoft (MSFT) — agents UiPath (PATH) — agents IBM (IBM) — agents

A model is a very large file of numbers, called weights, plus the architecture that says how to combine them. Text is cut into tokens, and the model predicts the next one, over and over. Its parameter count decides how much memory it needs just to exist, before it does any work.

Alphabet (GOOGL) — frontier lab Meta (META) — open weights Alibaba (BABA) — frontier lab

Serving software turns one model into something thousands of people can use at once. Each request has two phases. Prefill reads your whole prompt in parallel, which is heavy on math. Decode then writes the answer one token at a time, re-reading the model’s weights and a running notebook of the conversation, the KV cache, for every token. Serving software mostly batches requests and manages that cache so the hardware stays busy.

Amazon (AMZN) — operate Microsoft (MSFT) — operate Cerebras (CBRS) — inference

PyTorch and JAX turn “multiply these matrices” into instructions a chip can run. For Nvidia, that path runs through CUDA and its libraries, which developers have written against for nearly two decades. That software, and the habit around it, is a large part of Nvidia’s moat. Google’s TPUs use their own compiler stack instead.

Nvidia (NVDA) — CUDA Alphabet (GOOGL) — JAX / XLA Meta (META) — PyTorch origin

Memory sets the speed limit. HBM, high-bandwidth memory, is stacked right beside the GPU and holds the weights and the KV cache. Larger, slower DRAM attached to the CPU holds the overflow, and SSDs hold the files at rest. Decode is limited by how fast bytes reach the compute, not by how fast the compute is. Nvidia’s Rubin supports up to 288 GB of HBM4 and AMD’s MI455 up to 432 GB, and Micron has said its entire 2026 HBM4 capacity is sold out.

SK hynix (000660.KS) — make Samsung (005930.KS) — make Micron (MU) — make

The CPU is the manager. It runs the operating system, handles networking and storage, schedules work, and feeds data to the GPUs. It does very little of the model math. Designs like Nvidia’s Grace pair the CPU tightly with the GPU so the two can share memory quickly: a GB200 NVL72 rack holds 36 Grace CPUs alongside 72 Blackwell GPUs.

Intel (INTC) — design AMD (AMD) — design Nvidia (NVDA) — design Arm (ARM) — design

The GPU does the matrix multiplication, thousands of operations at once. Everything above it exists to keep it fed, and everything below it, from power to cooling to the building, exists to keep it running. Custom accelerators such as Google’s TPU do the same job with a narrower design.

Nvidia (NVDA) — design AMD (AMD) — design Broadcom (AVGO) — design TSMC (TSM) — make

Training reads and writes enormous amounts of data and is limited mostly by compute and by the network between GPUs. Inference is limited mostly by memory, because decode re-reads the weights for every token. That is why HBM, layer 6, has become some of the most contested silicon in the stack.