How one prompt travels through a giant AI model
An animated, self-contained guide to LLM inference parallelism: DP, TP, PP, EP, their combinations, and the attention designs (MHA, GQA, MQA, MLA) that make them work. Every number on this page links to its official source. Works offline (fonts fall back to system fonts without internet).
How to use this page
Read the tabs left to right. Each tab has three parts: plain-English explanation first, an animation you control (Replay, Next, Speed), and a "when to use it" summary with real model names and trade-offs.
Every claim that comes from a paper or official documentation carries a small blue link like this one vLLM docs ↗. Click it to open the original in a new tab.
Jargon is explained the first time it appears, and again in the glossary on the last tab. If a word is not explained, that is a bug in this page, not in you.
The one idea behind everything
The map
Models used as examples on this page
| Model | Type | Layers | Query heads | KV heads | Hidden size | Weights in BF16 | Source |
|---|---|---|---|---|---|---|---|
| Llama 3.1 8B | dense | 32 | 32 | 8 | 4096 | about 16 GB | config.json ↗ Llama 3 paper ↗ |
| Llama 3.1 70B | dense | 80 | 64 | 8 | 8192 | about 141 GB | Llama 3 paper, Table 3 ↗ |
| Llama 3.1 405B | dense | 126 | 128 | 8 | 16384 | about 812 GB | Llama 3 paper, Table 3 ↗ config.json ↗ |
| DeepSeek-V3 | MoE, 671B total / 37B active | 61 (first 3 dense) | 128 (MLA) | n/a (latent) | 7168 | about 1.3 TB BF16 (shipped in FP8, about 700 GB) | DeepSeek-V3 report ↗ config fields ↗ |
Weight sizes use the rule "2 bytes per parameter in BF16" (8B × 2 = 16 GB). Some community copies of the 405B config list 16 KV heads; the Llama 3 paper and Meta's official Instruct config say 8, which is what this page uses.
1 · Basics: the words you need
Read this tab once. Everything later builds on these seven ideas.
1.1 DNN (deep neural network) and "model"
A neural network is a very large list of numbers (called weights or parameters) plus a fixed recipe for multiplying your input by those numbers. "Deep" just means the recipe has many stages stacked on top of each other. An LLM (large language model) is a deep neural network whose input is text and whose output is a guess for the next word.
Analogy: a model is a very long factory assembly line. The weights are the settings on every machine. Running the line once on a piece of raw material is called inference. Adjusting the machine settings is training; this page is only about inference.
Sizes: "8B" means 8 billion parameters. Each parameter is stored as a 2-byte number in BF16 (brain floating point, 16 bits), so 8B parameters take about 16 GB of memory. FP8 halves that (1 byte each) with a small quality cost.
1.2 Token, prompt, prefill and decode
Text is chopped into tokens (word pieces; "Paris" is one token, "unbelievable" might be three). Your prompt is the list of input tokens.
Inference has two phases. Prefill: all prompt tokens go through the whole model at once, in parallel, producing the first output token. Decode: the model runs again for each new token, one at a time, because token 10 depends on token 9. Prefill is compute-heavy; decode is memory-bandwidth-heavy because the whole model's weights are re-read for every single token NVIDIA ↗.
Two latency words you will see everywhere: TTFT (time to first token, how long prefill takes) and TPOT (time per output token, how fast decode streams) DistServe ↗.
1.3 Layer, attention and MLP
One layer (also called a transformer block) does two jobs in sequence: attention (let each token look at the other tokens and gather what is relevant) and an MLP (multi-layer perceptron, also called FFN or feed-forward network: a plain two-matrix calculation applied to each token on its own). Small normalisation steps sit around them. Llama 3.1 70B stacks 80 such layers Llama 3 paper ↗.
Press Play to watch one token go through one layer.
1.4 Query, Key, Value and attention heads
Inside attention, every token is turned into three small vectors: a Query (Q: "what am I looking for?"), a Key (K: "what do I contain?") and a Value (V: "what information do I hand over if someone picks me?"). A token's Query is compared with every earlier token's Key; the best matches hand over their Values, weighted. This is the "scaled dot-product attention" of the original Transformer paper Attention is all you need ↗.
An attention head is one independent copy of this Q-K-V comparison with its own small weights. A layer runs many heads in parallel (Llama 3.1 70B: 64 heads per layer), each free to look for a different kind of relationship (grammar, names, position). This is multi-head attention (MHA). The outputs of all heads are glued together and mixed by one more matrix.
The head's vector width is called head_dim (128 in all Llama 3 models). hidden size = heads × head_dim is roughly true for the query side: 64 × 128 = 8192 for 70B.
1.5 KV cache: why decode needs memory
During decode, each new token must compare its Query with the Keys of all earlier tokens. Recomputing those Keys and Values every step would redo all the work. So the model caches the K and V vectors of every token, for every layer, for every KV head. This is the KV cache, and it grows with every token in every active conversation NVIDIA ↗.
Size per token = 2 (K and V) × layers × KV heads × head_dim × bytes. Try it:
Per token: 320 KiB · This context: 2.5 GiB · All requests: 2.5 GiB
Default values are Llama 3.1 70B. Change KV heads to 64 to see what the same model would cost without GQA (tab 2): 8× more. vLLM's PagedAttention stores this cache in small fixed-size blocks so that memory is not wasted on reserved-but-unused space PagedAttention paper ↗.
1.6 GPU, node, NVLink, cluster
A GPU is the chip that does the matrix maths; a datacentre GPU such as an NVIDIA H100 has 80 GB of memory (HBM). A node is one server, typically with 8 GPUs joined by NVLink, a very fast private link between GPUs in the same box. Nodes talk to each other over a slower network (InfiniBand or Ethernet). A cluster is many nodes. NVIDIA's GB200 NVL72 rack extends the NVLink domain to 72 GPUs, so a whole rack behaves like one big node NVIDIA GB200 NVL72 ↗.
Rule of thumb this page repeats often: anything chatty stays on NVLink inside a node; anything that only needs to send a small packet once can cross nodes.
1.7 Inference gateway (router, load balancer)
Users never talk to a GPU. They talk to an inference gateway: a front door that accepts the request, picks which model copy (replica) should serve it, forwards it, and streams the answer back. Good gateways route by load, by which replica already has your conversation's KV cache (prefix-cache-aware routing), and by region. vLLM's data-parallel mode and SGLang's Model Gateway are examples of this layer vLLM docs ↗ SGLang DP guide ↗.
Analogy: the receptionist at a hospital. You tell them what you need; they send you to the least busy doctor who already has your file.
1.8 LMCache and KV cache reuse
Two users often send the same long text (a shared system prompt, a document, the earlier turns of a chat). Without help, every replica would redo the prefill. Prefix caching inside one engine keeps the KV cache of recent prefixes so an identical prefix skips prefill. LMCache is a separate KV-cache layer that stores KV caches of any reused text (not only prefixes) in GPU, CPU memory or disk and shares them across engine instances, so each text is prefilled once. It reports 3 to 10× lower delay in multi-turn and RAG workloads when paired with vLLM LMCache docs ↗ LMCache GitHub ↗.
Why it belongs in a parallelism guide: when you have many DP replicas, a shared KV store is what lets a gateway send you to any replica without losing your conversation's cache.
1.9 Batching
Reading 141 GB of weights to produce one token is wasteful. Batching runs many users' tokens through the weights together, so the weights are read once per step for everyone. Continuous batching lets new requests join and finished ones leave at every decode step instead of waiting for the whole batch Orca (OSDI 22) ↗. Most parallelism trade-offs on this page assume a batch, because throughput is measured in tokens per second across all users.
2 · MHA, GQA, MQA, MLA: how many Keys and Values to keep
All four do the same job: let each token's Query look at other tokens. They differ in how many Key/Value heads are stored, which decides the KV cache size and the memory traffic during decode.
- MHA, multi-head attention: every query head has its own K and V head. Best quality, biggest cache Vaswani et al. 2017 ↗.
- MQA, multi-query attention: all query heads share one K and one V head. Smallest cache, some quality loss. Introduced by Shazeer in 2019 Shazeer 2019 ↗. Used by PaLM and Falcon-7B.
- GQA, grouped-query attention: query heads are split into G groups; each group shares one K and one V head. GQA-1 is MQA, GQA-H is MHA. Quality close to MHA, speed close to MQA Ainslie et al. 2023 ↗. Used by Llama 2 (34B, 70B), all Llama 3, Mistral, Qwen2 and later.
- MLA, multi-head latent attention: instead of storing K and V, store one small compressed "latent" vector per token and rebuild K and V from it on the fly. Cache even smaller than GQA, quality kept. Introduced in DeepSeek-V2, used in DeepSeek-V3 DeepSeek-V2 ↗ DeepSeek-V3 ↗.
Numbers for Llama 3.1 70B, 8192-token context, BF16
| Variant | KV heads | KV cache per token (80 layers) | KV cache for 8k tokens | Who uses it |
|---|---|---|---|---|
| MHA | 64 | 2.5 MiB | 20 GiB | GPT-2, original Llama 1, Llama 2 7B/13B |
| GQA | 8 | 320 KiB | 2.5 GiB | Llama 2 70B, Llama 3 (all), Mistral, Qwen2+, Falcon-40B |
| MQA | 1 | 40 KiB | 320 MiB | PaLM, Falcon-7B, StarCoder |
| MLA | latent, 512 + 64 numbers per token per layer in DeepSeek-V3 | about 70 KiB per token for 61 layers | about 560 MiB | DeepSeek-V2, V3, R1 |
Formula: 2 × layers × KV heads × head_dim × 2 bytes. 2 × 80 × 8 × 128 × 2 = 327,680 bytes = 320 KiB. MLA stores kv_lora_rank (512) plus the rotary part (64) per token per layer DeepSeek-V3 config fields ↗.
Why this matters for parallelism
3 · Data parallelism (DP): copy the whole model
What it is. Keep several complete copies (replicas) of the model, each on its own GPU or GPU group, and let a gateway send each request to one replica. Data parallelism replicates the entire model across multiple GPU sets and processes different batches of requests in parallel vLLM docs ↗.
Analogy. A bank opens more identical counters. Each customer still waits the same time at the counter, but the queue is shorter.
What moves between GPUs. Nothing, for a dense model. Each replica is independent. (For MoE models the replicas must step in lockstep so the expert layers can be shared; see tab 6.)
Journey below. Llama 3.1 8B (16 GB) on 2 nodes × 8 GPUs = 16 replicas. The gateway picks the least-loaded GPU; the prompt runs through all 32 layers on that GPU only.
When to use DP
| Use it when | Example | Gains | Costs / limits |
|---|---|---|---|
| The model already fits on one GPU (or one TP group) and you need to serve more users. | Llama 3.1 8B: one 80 GB GPU per replica; 16 replicas on 2 nodes. Mistral 7B, Qwen2.5 7B/14B, Gemma. | Throughput scales almost linearly with replicas. Shorter queues, so lower latency under load. | Does not make a single request faster. Every replica needs the full weights, so memory cost multiplies. Needs a gateway, and ideally a shared KV store (LMCache) so any replica can resume a chat. |
| On top of any other strategy, once one replica is as efficient as it will get. | Llama 3.1 405B: DP=2 of (PP=2 × TP=8). --data-parallel-size in vLLM. | The only strategy that scales to "global population": add replicas, racks, regions. | For MoE models with EP, DP ranks must run in lockstep; vLLM shards MoE layers across TP×DP vLLM docs ↗. |
4 · Tensor parallelism (TP): cut inside every layer
What it is. Every weight matrix in every layer is sliced across N GPUs. Attention is split by heads (GPU 0 gets heads 0–7, GPU 1 gets 8–15, ...), the MLP is split by columns then rows, and the vocabulary matrix is split too. Tensor parallelism shards model parameters across multiple GPUs within each model layer vLLM docs ↗. The scheme comes from Megatron-LM, which calls it intra-layer model parallelism Megatron-LM paper ↗.
Analogy. Eight cooks each chop one eighth of every vegetable for every dish. Fast, but after every chop they have to pass bowls around to combine the pieces.
What moves between GPUs. Partial results. Each layer needs two all-reduce operations (every GPU sends its partial sum to every other GPU and they all end with the total): one after attention, one after the MLP. For an 80-layer model that is 160 all-reduces per forward pass, and decode does one forward pass per token. This traffic is why TP lives on NVLink inside one node.
Important: TP does not cut layers across nodes. Every GPU holds a slice of every layer. Cutting between layers is PP (tab 5).
Journey below. Llama 3.1 70B (141 GB) on TP=8 inside one node, about 18 GB per GPU. Toggle the "bad case" to see the same TP=8 stretched across two nodes.
Layer stack: current · done · pending. Slice: Q heads · KV head · MLP. Links: NVLink · cross-node.
When to use TP
| Use it when | Example | Gains | Costs / limits |
|---|---|---|---|
| The model does not fit on one GPU but fits in one node. | Llama 3.1 70B at TP=4 or TP=8 on H100s; Qwen2.5 72B; Mixtral 8x22B. vLLM: --tensor-parallel-size 8 vLLM docs ↗. | Lowest latency of all strategies: N GPUs work on the same token at once, so each token is produced faster. KV cache is also split, so longer contexts fit. | NVIDIA describes tensor parallelism as a higher-overhead mechanism for reducing latency, versus pipeline parallelism as a low-overhead way to raise throughput NVIDIA blog ↗. Across nodes without NVLink it becomes the slowest option. TP beyond the KV-head count (8 for Llama 3) duplicates KV heads. |
| Whole rack is one NVLink domain. | GB200 NVL72: TP can span the 72-GPU rack for trillion-parameter models NVIDIA ↗. | Node boundary stops mattering. | Hardware cost. |
5 · Pipeline parallelism (PP): cut between layers
What it is. Give layers 1–20 to GPU A, 21–40 to GPU B, and so on. Each GPU is a stage. A request flows stage to stage like a car on an assembly line. Pipeline parallelism distributes model layers across multiple GPUs vLLM docs ↗. The idea comes from GPipe GPipe paper ↗.
What moves between GPUs. Only the activations at the cut: the hidden-state vector (8192 numbers per token for 70B) plus the residual stream. That is one small packet per token per stage boundary, so PP can cross the node boundary cheaply.
The catch: the bubble. While stage 1 works, stage 0 is idle unless another request is behind it. The fix is micro-batching: split the batch into chunks so every stage always has something to chew on.
Journey below. Llama 3.1 70B, PP=4 with TP=1 so the mechanics are visible: 4 stages of 20 layers, about 35 GB each, two stages per node.
When to use PP
| Use it when | Example | Gains | Costs / limits |
|---|---|---|---|
| The model does not fit in one node, or nodes have no NVLink between them. | Llama 3.1 405B on 2 nodes: TP=8 inside, PP=2 across. vLLM: --tensor-parallel-size 8 --pipeline-parallel-size 2; the common practice is TP = GPUs per node, PP = number of nodes vLLM docs ↗. | Fits bigger models with little network traffic. Raises throughput; NVIDIA measured a 1.2× MLPerf gain from PP on Llama 2 70B NVIDIA blog ↗. | Does not reduce single-request latency and can add a little. Needs enough concurrent requests (micro-batches) to hide the bubble. Uneven stages waste GPUs. |
6 · Expert parallelism (EP): spread the experts
First, what is MoE? In a mixture-of-experts model the MLP of each layer is replaced by many small MLPs called experts, plus a tiny router that picks a few experts for each token. DeepSeek-V3 has 256 routed experts and 1 shared expert per MoE layer and activates 8 routed experts per token, so only 37B of its 671B parameters are used for any given token DeepSeek-V3 report ↗. The idea comes from GShard and Switch Transformer at Google GShard ↗ Switch Transformer ↗.
Analogy. A hospital with 256 specialists. Each patient (token) sees only 8 of them. Total expertise is huge; cost per patient is small. But you must walk the patient to each specialist's room.
What EP is. Put different experts on different GPUs instead of slicing every expert across all GPUs (which is what TP would do). With EP=8 each GPU stores 32 of the 256 experts for every MoE layer. vLLM: expert parallelism replaces tensor parallelism for the MoE layers and uses the same degree as TP vLLM docs ↗. SGLang and vLLM pair it with DP attention: each GPU runs attention and holds the KV cache for its own requests, only the expert layer is shared SGLang DPA guide ↗ vLLM docs ↗.
What moves between GPUs. The token itself. An all-to-all sends each token's hidden state to the GPUs that hold its chosen experts (dispatch), then a second all-to-all brings the 8 results back (combine). Two all-to-alls per MoE layer.
Journey below. DeepSeek-V3 on one 8-GPU node, EP=8 with DP attention. Which 8 experts fire is illustrative; real routing differs per token.
When to use EP
| Use it when | Example | Gains | Costs / limits |
|---|---|---|---|
| The model is MoE and the experts are the bulk of the weights. | DeepSeek-V3/R1 (256 experts), Qwen3-235B-A22B, Mixtral 8x22B, Llama 4 Maverick. vLLM: --enable-expert-parallel; SGLang: --enable-dp-attention --tp 8 --dp 8 SGLang DeepSeek guide ↗. | Experts are stored once, not copied, so a 671B model fits. Each GPU's experts serve tokens from every request, so big batches keep every expert busy. SGLang reports up to 1.9× decode throughput from DP attention on DeepSeek SGLang ↗. | All-to-all traffic every MoE layer. Expert load imbalance (some experts popular) wastes GPUs; needs balancing (EPLB). DeepSeek trained with 64-way EP across 8 nodes and no TP at all DeepSeek-V3 report ↗. Not for small-batch, low-latency serving. |
7 · Combinations: how frontier models are actually served
Real deployments stack the strategies. The ordering rule from vLLM: tensor parallel inside the node, pipeline parallel across nodes, then data parallel on top vLLM docs ↗. Megatron-LM's follow-up paper gives the same rule for training: TP within a server, PP across servers, DP to scale out Narayanan et al. 2021 ↗. Llama 3 405B was trained with 4D parallelism: TP, PP, context parallelism and data parallelism Llama 3 paper ↗.
7.1 TP + PP (one replica of a model too big for a node)
TP=8 inside each node, PP = number of nodes. The journey: the prompt is broadcast to the 8 GPUs of the first node, every layer of that node's stage runs sharded with 2 all-reduces, then the activations cross once to the next node and the pattern repeats. Llama 3.1 405B in BF16 needs 812 GB, more than a node of 8 × 80 GB, so PP=2 × TP=8 (16 GPUs, about 51 GB each) is the minimum in BF16; in FP8 a single TP=8 node fits.
7.2 TP + DP (model fits a node, you need more throughput)
Each replica is one TP group; the gateway spreads requests over replicas. Llama 3.1 70B: TP=8 per node, DP=4 across four nodes = 32 GPUs serving four independent streams. vLLM: --tensor-parallel-size 8 --data-parallel-size 4. Nothing crosses nodes at all.
7.3 TP + PP + DP (the 405B journey)
7.4 DP attention + EP (+ TP) for MoE frontier models
For DeepSeek-class models the inner pattern changes: attention is data-parallel (each GPU keeps its own requests and KV cache), the expert layers are expert-parallel across the whole group, and TP is used only where it still helps (dense layers, or attention when batches are small). SGLang shows both shapes: --enable-dp-attention --tp 8 --dp 8 on one node, or --tp 16 --dp 2 across two nodes, meaning two DP groups of 8 TP GPUs SGLang ↗. vLLM shards MoE layers across TP × DP when EP is on vLLM ↗. Large deployments run EP across many nodes (SGLang documents a 96-GPU H100 recipe with prefill/decode disaggregation and large-scale EP) SGLang ↗.
7.5 Other dimensions you will meet
- Context parallelism (CP) / sequence parallelism. Split a very long prompt's tokens across GPUs so prefill of a 128k-token input fits and runs in parallel. Used by Llama 3 for long-context training Llama 3 paper ↗; SGLang exposes
--attn-cp-sizefor DeepSeek V3.2 prefill SGLang ↗. - Prefill/decode (PD) disaggregation. Run prefill on one GPU pool and decode on another, each with its own parallelism, and ship the KV cache between them. DistServe showed this removes prefill-decode interference and raises goodput under latency targets DistServe (OSDI 24) ↗; vLLM, SGLang and NVIDIA Dynamo ship it, often with LMCache or NIXL moving the cache NVIDIA Dynamo docs ↗.
- Global scale. Everything above is one "unit". Serving a planet is DP all the way up: many units per cluster, many clusters per region, a global router choosing the region. No new parallelism is needed, only more copies and smarter routing.
7.6 Try it: a parallelism planner
This is the vLLM rule of thumb made into arithmetic: pick the smallest TP that fits (never more than GPUs per node, and warn beyond the KV-head count), add PP stages if one node is not enough, and leave the rest of the memory for KV cache. Real planning also considers batch size, context length and your latency target.
8 · When to use what
Decision flow
The full comparison
| Strategy | What is cut | What moves, how often | Latency per request | Throughput | Memory per GPU | Best placed | Typical models |
|---|---|---|---|---|---|---|---|
| DP | nothing; copies | nothing (dense) | unchanged; queues shorter | linear in replicas | full model | anywhere | 8B–14B dense; the outer layer of every big deployment |
| TP | inside every layer (heads, MLP columns) | 2 all-reduces per layer per forward pass | lower (N GPUs per token) | up, with overhead | weights/N, KV/N up to KV-head count | inside a node on NVLink (or NVL72) | Llama 70B TP=4/8; 405B TP=8; Mixtral |
| PP | between layers (stages) | activations once per stage boundary | same or slightly higher | up, needs micro-batches | weights/stages | across nodes | 405B PP=2; any model on PCIe-only boxes |
| EP | the expert set of MoE layers | 2 all-to-alls per MoE layer | depends on fabric | high with big batches | experts/N, attention replicated | inside node or fast fabric; wide EP across nodes for DeepSeek-class | DeepSeek-V3/R1, Qwen3-MoE, Llama 4 |
| CP | the prompt's tokens | K/V exchange during attention | prefill of long prompts faster | neutral | KV/N for that request | inside node | 128k-token prefill, DeepSeek V3.2 NSA prefill |
Trade-off summary in one sentence each
DP buys capacity with memory. TP buys latency with interconnect bandwidth. PP buys model size with pipeline bubbles. EP buys parameter count with all-to-all traffic and balancing headaches. CP buys prompt length with attention communication. Every production system picks a mix, and the mix changes with the hardware: NVIDIA states that pipeline parallelism is a low-overhead way to raise throughput while tensor parallelism is a higher-overhead way to cut latency NVIDIA blog ↗.
9 · Bonus: the natural next steps
Once the four cuts make sense, these are the topics you will meet next, in a sensible order. Each has one primary source.
PagedAttention and vLLM
Store the KV cache in fixed-size blocks like virtual memory pages so nothing is reserved and unused. The paper behind vLLM arXiv 2309.06180 ↗.
Continuous batching
Let requests join and leave the batch at every decode step. Orca introduced it Orca, OSDI 22 ↗.
Speculative decoding
A small draft model guesses several tokens; the big model checks them in one pass, keeping the exact output distribution. Two independent papers Leviathan et al. ↗ Chen et al. (DeepMind) ↗. DeepSeek-V3's multi-token prediction heads are used the same way in SGLang SGLang ↗.
Prefill/decode disaggregation
Separate GPU pools for the two phases, each with its own parallelism, KV cache shipped between them DistServe ↗. Productised in vLLM, SGLang and NVIDIA Dynamo Dynamo docs ↗.
Quantization: FP8, INT4
Store weights and KV cache in fewer bits to halve memory and bandwidth. Llama 3.1 405B is commonly served in FP8 so one node fits vLLM quantization docs ↗.
FlashAttention
Compute attention without materialising the full score matrix, so long contexts fit and run faster FlashAttention ↗.
Expert load balancing (EPLB)
Popular experts get duplicated, unpopular ones consolidated, so EP GPUs stay evenly busy. DeepSeek-V3's auxiliary-loss-free balancing and SGLang's EPLB DeepSeek-V3 ↗ SGLang EP ↗.
Running on your own machine: llama.cpp
Everything above shrinks to one laptop with aggressive quantization (GGUF) and CPU/GPU offload. Good for understanding one layer at a time llama.cpp ↗.
How training parallelism differs
Training adds gradients and optimizer state, so DP becomes FSDP/ZeRO (shard those too) and PP needs clever schedules. Megatron's combined-parallelism paper and Llama 3's 4D parallelism are the references Megatron 2021 ↗ Llama 3 ↗.
Rack-scale NVLink
GB200 NVL72 turns 72 GPUs into one NVLink domain, moving the "TP stays in the node" boundary to the rack NVIDIA ↗.
Sources
All links open in a new tab. Primary sources only: papers on arXiv, vendor documentation, official model configs.
Glossary (every abbreviation on this page)
- Activation / hidden state
- The vector of numbers that represents a token as it passes between layers. For Llama 3.1 70B it has 8192 numbers.
- All-reduce
- A collective operation where every GPU in a group contributes a partial result and every GPU ends with the sum. Used twice per layer in TP.
- All-to-all
- A collective where every GPU sends different data to every other GPU. Used to dispatch tokens to experts and bring results back in EP.
- Attention head
- One independent Q/K/V comparison inside the attention block. A layer runs many in parallel.
- BF16 / FP16 / FP8 / INT4
- Number formats using 16, 16, 8 and 4 bits. Fewer bits = less memory and bandwidth, slightly less precision.
- Batch, micro-batch
- A group of requests processed together. A micro-batch is a slice of a batch fed into a pipeline so stages stay busy.
- Bubble
- Idle time on a pipeline stage waiting for work.
- Cluster
- Many nodes connected by a network.
- CP (context parallelism)
- Splitting one request's tokens across GPUs.
- Decode
- The phase that produces output tokens one at a time.
- Dense model
- A model where every parameter is used for every token (Llama 3). Opposite of MoE.
- DNN (deep neural network)
- Many layers of weighted calculations stacked; an LLM is one.
- DP (data parallelism)
- Several full copies of the model serving different requests.
- DP attention (DPA)
- Data parallelism applied only to the attention block, used with EP for MoE models.
- EP (expert parallelism)
- Different experts of an MoE layer placed on different GPUs.
- EPLB
- Expert parallelism load balancer: duplicates hot experts so GPUs stay evenly busy.
- Expert
- One of the small MLPs in an MoE layer. A router picks a few per token.
- FFN / MLP
- Feed-forward network / multi-layer perceptron: the non-attention half of a layer.
- FSDP / ZeRO
- Training-time data parallelism that also shards weights, gradients and optimizer state.
- Gateway (inference gateway, router, load balancer)
- The service that receives user requests and chooses which replica serves them.
- GPU
- Graphics processing unit; the chip that runs the model. H100: 80 GB memory.
- GQA / MQA / MHA / MLA
- Grouped-query, multi-query, multi-head, multi-head latent attention: variants that keep fewer K/V heads to shrink the KV cache. See tab 2.
- HBM
- High-bandwidth memory on the GPU, where weights and KV cache live.
- head_dim
- Width of one attention head's vectors (128 in Llama 3).
- Hidden size
- Width of the activation vector (4096 / 8192 / 16384 for Llama 3.1 8B / 70B / 405B).
- InfiniBand / RoCE / Ethernet
- Networks that connect nodes. Much slower than NVLink.
- Inference
- Running a trained model to get outputs. This page is only about inference.
- KV cache
- Stored Key and Value vectors of earlier tokens, kept so decode need not recompute them.
- Layer (transformer block)
- Attention + MLP + normalisation. Models stack 32 to 126 of them.
- LLM
- Large language model.
- LMCache
- A KV-cache storage and sharing layer that lets any text be prefilled once and reused across engines.
- MoE (mixture of experts)
- A model whose MLP is replaced by many experts and a router; only a few experts run per token.
- Node
- One server, usually 8 GPUs joined by NVLink.
- NVLink / NVSwitch
- NVIDIA's fast GPU-to-GPU link inside a node (or an NVL72 rack).
- NVL72
- NVIDIA GB200 rack where 72 GPUs share one NVLink domain.
- PagedAttention
- vLLM's block-based KV cache memory management.
- Parameter / weight
- One learned number in the model. 8B = 8 billion of them.
- PD disaggregation
- Running prefill and decode on separate GPU pools.
- PP (pipeline parallelism)
- Different layers on different GPUs (stages).
- Prefill
- The phase that processes all prompt tokens at once and produces the first output token.
- Prefix caching
- Reusing the KV cache of a prompt prefix that was seen before.
- Q / K / V (query, key, value)
- The three vectors each token is turned into inside attention.
- RAG
- Retrieval-augmented generation: putting retrieved documents into the prompt.
- Replica
- One complete, independently serving copy of the model (may itself span several GPUs).
- Residual stream
- The running sum that each layer adds its output to; it is what PP sends between stages.
- Router (in MoE)
- The tiny network that scores experts and picks the top-k for each token.
- Shared expert
- An expert every token always uses, in addition to the routed ones (DeepSeek design).
- Speculative decoding
- Guess several tokens with a small model, verify with the big one.
- Stage
- One GPU's share of layers in PP.
- Throughput
- Tokens (or requests) per second across all users.
- Token
- A word piece; the unit the model reads and writes.
- TP (tensor parallelism)
- Every layer's weights sliced across GPUs.
- TTFT / TPOT
- Time to first token / time per output token: the two latency numbers users feel.
- vLLM / SGLang / TensorRT-LLM / llama.cpp
- Inference engines. The first three are datacentre servers; llama.cpp targets laptops and single machines.