Next-Concept Modeling (NCM), explained with animations
How a language model can write a whole paragraph in ~12 parallel steps instead of hundreds of sequential ones: a learned coarse-to-fine hierarchy of discrete codes, generated one level at a time. Start with Tab 0 if you have never met an LLM or a diffusion model. Every number here is from the paper, blog or repo linked at the bottom of each card.
Foundations · What is an LLM, what is a diffusion model, and where does NCM sit?
Three ways a machine can write a paragraph. Think of three writers: a typist who commits one word at a time, an editor who rewrites the whole draft many times, and an architect who sketches an outline first and fills in the words last.
Three writers, one paragraph
The three columns are the three model families described below. The counter under each is the number of times the full neural network has to run. Analogy adapted from "NCM: Is this the End of LLMs?" (Kai, Oct 2026): "sketch first, words last".
The pyramid: coarse to fine, each level adds only what's missing
The single most useful mental model for NCM. Diffusion refines a flat canvas of L slots on every pass. NCM builds a pyramid: level 1 is a handful of codes for the whole text, each level below is wider, the bottom level is the full L positions. A level never restates what the levels above already said; it encodes only the residual, the detail still missing. The text under the pyramid is the paper's own German→English sample decoded after the matching levels.
Same checklist for both: many positions at once ✓, several passes ✓, faster than word-by-word ✓. The difference is structure: flat vs pyramid. Pyramid widths here are 2, 4, 6, … 16 for legibility; the paper's real schedules start at 1 code and end at L (Tab 2). Visual metaphor after Kai's video; text rows from paper Table 12.
1. What is a Large Language Model (LLM)?
- Neural network
- A big function with billions of adjustable numbers (parameters or weights) that maps an input to an output. "Training" means nudging those numbers so the outputs match examples.
- Transformer
- The network architecture behind essentially every modern language model (Vaswani et al. 2017). Its core operation, attention, lets every position in the text look at every other position and decide what is relevant.
- Language model
- A network trained on text to output a probability for what comes next. "Large" means billions of parameters trained on trillions of tokens. GPT, Claude, Gemini, Llama, Qwen are all LLMs.
- Autoregressive (AR) generation
- How an LLM writes: predict a probability distribution over the next token, pick one, append it, repeat. One token per forward pass;
Ltokens needLsequential passes. The model only ever looks left (causal attention), which is what makes the KV cache possible: past tokens' attention keys/values are stored, so each new step only computes the new token. - Why it's slow on your laptop
- Every one of those
Lpasses must stream the entire set of weights from memory through the chip. A 16 GB model writing 1,024 tokens moves ~16 TB of memory traffic. At small batch sizes, decoding is bound by memory bandwidth, not by arithmetic. (This is the "2 AM blinking cursor" the video opens with.)
L sequential steps, and no way to revise a word once it's written.2. What is a diffusion model, and a diffusion language model?
- Diffusion (images)
- Train a network to remove noise. Take a photo, add Gaussian noise step by step until it is pure static, and teach the network to reverse each step. To generate, start from static and denoise
Ttimes. Stable Diffusion, DALL·E, Imagen work this way (Ho et al. 2020). - Discrete / masked diffusion (text)
- Text isn't continuous, so the "noise" is masking: replace tokens with a blank
[MASK]. Start from an all-masked canvas ofLblanks; at each ofTsteps the network predicts every blank in parallel and commits to some of them. D3PM, SEDD, MDLM, LLaDA, and the commercial Mercury and DiffusionGemma. - Continuous / flow text models
- Variants that add Gaussian noise to token embeddings instead of masking (CDCD, CANDI, ELF, AURORA-LM, flow-map LMs). Same shape:
Tpasses over the whole canvas. - Bidirectional attention
- Because every position is being refined at once, each position must look both left and right. That is what lets diffusion revise earlier words, and also what breaks the AR-style KV cache.
- Why it's not automatically faster
- The promise is
T ≪ L. In practice quality collapses whenTis small, so published models useT= 64–1,024 steps, each step processing allLpositions. The editor rewrites the whole draft, many times.
3. How Next-Concept Modeling differs from both
NCM keeps diffusion's parallelism and ability to revise, but replaces "refine the whole canvas T times" with "build a hierarchy coarse-to-fine in K levels":
- Different representation. AR and diffusion both operate directly on tokens. NCM first learns a tokenizer (a multiscale residual VQ-VAE, Tab 2) that turns
Ltokens intoKlevels of integer codes: 1 code for the gist, then 2, 4, … up toLfor the details. The coarse levels are what the paper calls concepts. - Different generation order. AR goes left-to-right. Diffusion goes "everything, blurry → everything, sharp". NCM goes outline → detail: level 1 (one code) fixes the sentence's shape, level 2 splits it into halves, and so on. Every code within a level is sampled in one parallel pass.
- Different cost curve. AR:
Lcheap steps. Diffusion:Tsteps each costingL. NCM:Ksteps costing 1, 2, 4, …L; the first half of the steps are almost free. The weights are readK≈ 12–32 times instead ofL= 128–1,024 times, which moves the bottleneck from memory bandwidth toward compute. - Different training. Two stages (tokenizer, then generator), but the generator trains on all
Klevels in one forward pass with a block-diagonal mask (Tab 3), and ends up needing ~4–5× fewer training tokens than diffusion baselines. - Same transformer. The generator is a standard 12-block, 768-wide transformer. Nothing exotic in the architecture; the novelty is what it predicts and in which order.
| LLM (autoregressive) | Diffusion LM | Next-Concept Model | |
|---|---|---|---|
| Writer analogy | Typist: one word, never look back | Editor: rewrite the whole draft T times | Architect: outline once, details once |
| Unit predicted | next token | all masked tokens | all codes of the next level |
| Sequential network calls | L (128–1,024) | T (64–1,024) | K (8–32) |
| Work per call | 1 position (+ cache) | L positions | ℓk positions, growing 1 → L |
| Can revise? | no | yes | yes (finer levels correct coarser) |
| Output length | variable | fixed canvas | fixed L |
| Maturity | production, trillions of tokens | production (Mercury, DiffusionGemma), billions of params | research, ≤725M params, Oct 2026 |
Sources: NCM paper; Kai, "NCM: Is this the End of LLMs?" for the writer analogy and the memory-bandwidth framing.
Why not one token at a time?
Three ways to produce a sentence of L tokens. The animation below races them on the same 16-token sentence. Watch two counters: sequential steps (wall-clock) and positions processed (compute).
Glossary first (hover-free, everything inline)
- Token
- The unit a text model reads and writes: a word or word-piece. "medium-sized" may be 3 tokens.
Lis the number of tokens in the output. - Autoregressive (AR) / next-token prediction (NTP)
- Generate token 1, feed it back, generate token 2, …
Lsequential steps. GPT-style models. - KV cache
- AR models store each past token's attention Keys and Values so each new step only processes the new token. Needs causal attention (left-to-right).
- Diffusion language model (DLM)
- Start from a fully masked (or noisy) sequence, and refine all
Lpositions in parallel forTdenoising steps. Bidirectional attention, so no AR-style KV cache. Examples: MDLM, DUO, LLaDA, Mercury. - NFE
- Number of Function Evaluations: how many times the big neural network is called to produce one output. AR:
L. DLM:T. NCM:K(number of levels). - Concept (in NCM)
- Not a philosophical object. The paper defines it as "the information carried by the coarse levels of the hierarchy": a short code sequence that fixes the gist before details exist.
- Level
k, lengthℓk - NCM represents text as
Klevels of discrete codes, levelkhavingℓkcodes, withℓ1 < ℓ2 < … < ℓK = L. - Generative perplexity (Gen. PPL)
- How "surprised" a strong reference model (GPT-2-large) is by generated text. Lower = more fluent. Can be gamed by producing boring text, so always read it next to…
- Unigram entropy (H)
- Word-level diversity of a sample. Real LM1B text ≈ 4.31. A model with H well below the data is repeating itself.
- BLEU / ROUGE
- Overlap scores vs a human reference: BLEU for translation, ROUGE-1/2/L for summarization. Higher is better.
Three paradigms race on 16 tokens
NTP: L = 16 steps, 1 new position each. Masked diffusion: here T = 8 steps, all 16 positions every step (128 position-evaluations). NCM: K = 5 levels of 1, 2, 4, 8, 16 codes (31 position-evaluations), then one decoder pass. Level lengths are illustrative; real schedules in Tab 2.
What's wrong with diffusion LMs in practice
The paper's own diagnosis of why DLMs haven't delivered the speed they promise:
- Many sampling steps. Cut
Tand quality collapses (their Fig. 5), so the parallelism doesn't buy much. - Expensive steps. Every step processes the full length-
Lsequence; bidirectional attention blocks the AR KV cache. Block-diffusion gets the cache back but gives up parallelism between blocks. - Huge training budgets. MDLM and DUO train on OpenWebText for 1M steps, >500B tokens. Distillation to few steps adds yet another training stage and reduces diversity.
Source: NCM paper §1.
The NCM bet
Change both the representation and the generation order:
- Represent text as a hierarchy of discrete codes, short-and-coarse to long-and-fine (a multiscale residual VQ-VAE; Tab 2).
- Generate the hierarchy level by level with one transformer; inside a level, sample all
ℓkcodes in parallel (Tab 3). - Early steps process 1, 2, 4… positions, so they are nearly free.
K ≪ L: 12 steps for 128 tokens, 32 for 1,024. - No distillation, no second training stage, fixed generation order (which the authors suspect is simply easier to learn than diffusion's "any order").
Borrowed from images: VAR's "next-scale prediction" (Tian et al. 2024) and residual quantization from audio codecs (SoundStream). The hard part for text: there is no natural "downsample" of a sentence, and one wrong token can flip meaning.
Stage 1 · A multiscale residual VQ-VAE tokenizer
Goal: turn L tokens into K levels of integer codes such that decoding only the first k levels already gives a rough version of the text, and all K levels give it back exactly. Trained first, then frozen forever.
Jargon for this stage
- Latent
z - The encoder's output: one
d-dimensional vector per token position, soz ∈ ℝL×d(d= 384 in the paper). In the animation each vector is drawn as a single bar height. - Vector quantization (VQ)
- Replace a continuous vector by the index of its nearest entry in a learned table called the codebook. The index is the discrete code. VQ-VAE.
- Residual quantization (RQ)
- Quantizing once leaves an error (the residual). Quantize the residual with a second codebook, add it on, repeat. Each level corrects the previous one.
- Multiscale RQ
- Same idea, but level
kfirst downsamples the residual toℓkvectors (Downk), quantizes those, then upsamples the picked codebook vectors back to lengthL(Rk) before adding. Early levels are short. - Cumulative reconstruction
ẑ(k) - Sum of the first
klevels' contributions.ẑ(0) = 0,ẑ(K) ≈ z. This is what the generator conditions on and what the decoder reads. - Nested dropout
- During training, randomly keep only levels 1…k and ask the decoder to reconstruct from that. Forces early levels to carry meaning instead of dumping everything into the fine levels.
- Straight-through estimator
- argmin has no gradient. Trick: in the backward pass, pretend the quantizer was the identity so the encoder still learns.
- Commitment loss
- MSE term that keeps encoder outputs close to codebook entries (and codebook entries close to encoder outputs) so quantization stays stable.
Encoding 8 tokens into 4 levels (1, 2, 4, 8 codes)
zcumulative reconstruction ẑ(k)residual z − ẑ(k)chosen codebook entryToy numbers: latents are 1-D (real ones are 384-D), codebooks have 6 entries (real: 4,096–8,192), Downk is average pooling and Rk is nearest-neighbour upsampling (the paper describes these as learned/resize operators; the exact form is in the code). The mechanics, residual shrinking level by level, are exact.
Interactive · Nested dropout: what do the first 1 level(s) alone carry?
The residual energy left after k levels (right) is what nested dropout attacks: because training sometimes cuts the hierarchy at a random k, the model is penalised if ẑ(k) alone decodes badly. The paper sets the drop probability to 0.5 for most datasets (LM1B: full vs truncated hierarchy weighted 0.7/0.3).
The equations, with a plain-English reading of each
Actual tokenizer configurations (Appendix C.1)
| Dataset | Target L | Levels K | Codebook |𝒱k| | d | Enc / Dec blocks | Params | Recon. acc. | Utilization |
|---|---|---|---|---|---|---|---|---|
| OpenWebText | 1,024 | 32 | 4,096 | 384 | 0 / 1 | 111.2M | 99.99% | 83.1% |
| LM1B | 128 | 12 | 8,192 | 384 | 0 / 4 | 111.4M | 99.86% | 89.3% |
| TinyStories | 128 | 12 | 4,096 | 384 | 0 / 1 | 36.4M | 99.98% | 90.1% |
| XSum | 48 | 8 | 4,096 | 384 | 0 / 1 | 60.6M | 99.79% | 86.2% |
| WMT14 De–En | 64 | 11 | 4,096 | 384 | 0 / 1 | 144.0M | 98.79% | 61.9% |
| Sudoku | 89 | 12 | 16 | 256 | 0 / 1 | 14.9M | 100.00% | 80.7–83.9% |
Reconstruction accuracy = exact token recovery on held-out text using all K levels. Utilization = mean % of codebook entries actually used. Note the WMT14 tokenizer's lower utilization (61.9%) and accuracy (98.79%): the 151k-token Qwen3 text vocabulary is harder to cover. Typical errors (Appendix G): swapped named entities ("Sydney → Melbourne"), numerals, diacritics, usernames. Sources: paper Tables 1, 5.
Stage 2 · The next-concept transformer
With the tokenizer frozen, train a plain transformer to predict level k's codes from the cumulative reconstruction of levels 1…k−1. The trick that makes training cheap: all K levels are predicted in one forward pass with a block-diagonal attention mask.
Jargon for this stage
- Teacher forcing
- During training, feed the model the true previous levels (from the tokenizer), not its own guesses. Fast, but creates a train/test mismatch.
- Conditioning corruption
- NCM's fix for that mismatch: randomly replace a fraction of the input codes with other codes from the same codebook; keep the targets clean. Teaches the model to recover from its own earlier mistakes. Rate decreases linearly with level; mean ≈ 0.5 in ablations.
- Block-diagonal attention
- Input positions for level
kmay attend only to other inputs of levelk. Why is that enough? Because the input for levelkisẑ(k−1), which already sums everything from earlier levels. - Prefix-LM
- For conditional tasks (translation, summarization, Sudoku) the prompt sits in front as a prefix. Prompt tokens attend to each other; each target block attends to the prompt + itself; the prompt never looks at targets.
- Classifier-free guidance (CFG)
- Run the model with and without the prompt and extrapolate the logits by a weight
w. Costs 2 NFE per step. Helps BLEU (peak at w=1.0), barely helps ROUGE (peak w=0.5). - MBR decoding
- Minimum Bayes Risk: sample
Kcandidates, keep the one most similar to all the others. Since NCM samples are cheap, you can afford 16 candidates in the NFE budget one diffusion sample costs. - RoPE
- Rotary position embeddings; the generator uses base 104–105.
One training step, all levels at once (K = 4, ℓ = 1, 2, 4, 8)
Inference: K sequential steps, each one fully parallel inside
This is Algorithm 2 of the paper. NFE = K generator calls + 1 decoder call. Every code inside a level is sampled independently given ẑ(k−1) (eq. 5); the coherence comes from the conditioning, not from attention between sampled codes.
The loss, and the one knob in it
Prefix-LM mask for conditional generation
Rows = queries, columns = keys. The prompt (P positions) is embedded with the frozen VQ-VAE encoder's token table and projected to the hidden width; the authors tried a frozen T5-small encoder instead and gained only +0.1–0.24 ROUGE for +35M parameters, so they kept their own (Appendix F.2.2).
Generator configurations (Appendix C.2)
| Dataset | Params | Blocks | Width | Heads | Prefix P | Updates | Batch | Loop structure |
|---|---|---|---|---|---|---|---|---|
| OpenWebText | 94.65M | 12 | 768 | 12 | 1 | 250k | 256 | unconditional, 32 levels |
| LM1B | 97.07M | 12 | 768 | 12 | 1 | 100k | 512 | unconditional, 12 levels |
| TinyStories | 88.61M | 12 | 768 | 12 | 1 | 50k | 2,048 | unconditional, 12 levels |
| XSum | 88.02M | 12 | 768 | 12 | 512 | 350k | 512 | prefix-LM, 8 levels |
| WMT14 | 119.96M | 12 | 768 | 12 | 96 | 150k | 2,048 | prefix-LM, 11 levels |
| Sudoku | 25.46M | 8 | 512 | 8 | 91 | 50k | 1,024 | prefix-LM (puzzle), 12 levels |
A GPT-2-small-shaped network. AdamW, peak LR 3e-4, BF16, torch.compile, seed 42. XSum/WMT14 generators are first trained on unpaired text, then on source–target pairs. Training-token accounting for LM1B: 25,957×2,048×128 (tokenizer) + 100,000×512×128 (generator) ≈ 13.36B positions vs 66B for the baselines.
Watch a generation refine, level by level
This is a real NCM run from the paper (Appendix J.2, Table 12): one German sentence, translated with the 11-level WMT14 model. After each level, the cumulative reconstruction is decoded to text and scored with BLEU against the reference.
German: Es besteht allerdings die Sorge, dass kleine und mittlere Unternehmen angreifbar gegenüber Hacking und Überwachung bleiben.
Reference: However, there is concern that small and medium-sized companies remain vulnerable to hacking and surveillance.
Decoded output after each of the 11 levels
Words highlighted in teal appear in the reference (case/punctuation-insensitive word match; the paper uses the same shading). Note two things the paper itself points out: (1) the skeleton ("However, … is … to … and …") is fixed at level 1 with a single code; (2) intermediate quality is not monotonic: BLEU dips at levels 3 and 9 before the last level fixes everything. Later levels correct earlier ones; that is the residual structure doing its job.
Why this matters for reasoning (Sudoku)
An AR model commits to cell 1, then cell 2, … and can never revise. A diffusion model can revise but spends 128 full-board passes. NCM refines a representation of the whole board across 12 levels, so the later levels can repair constraint violations introduced earlier. Result: 86.1% of hard boards solved at 12 NFE vs 81.3% for the best diffusion baseline at 16 NFE and ≈1% for AR at 128 NFE (Tab 5).
Appendix H probes the same idea on TinyStories: fix the first 3 of 12 levels and resample the rest → same plot, different names and objects. Mix the coarse codes of two stories → a blended story. Swap just 2 of level 1's codes → characters and objects change while the narrative shape survives. Coarse levels really do hold the "concept".
Results and trade-offs
All NCM generators are ~100M parameters (25M for Sudoku). Baselines are at comparable or larger sizes. Bold = best in its group. Everything below is copied from the paper's tables; nothing is re-estimated.
Wall-clock, measured on one H100 SXM 80GB (1,024 sequences, batch 16)
Speedups are relative to the strongest diffusion baseline on each task (DUO on LM1B, AURORA-LM on XSum). Two sources of speed: fewer NFE (11× fewer than DUO, 16× fewer than AURORA-LM) and cheaper early steps because coarse levels are short. Source: blog, paper §4.6.
Unconditional generation (paper Table 8; Gen. PPL under GPT-2-large, lower is better; H = unigram entropy, closer to Data is better)
| Method | TinyStories NFE | Gen. PPL | H | LM1B NFE | Gen. PPL | H | OWT NFE | Gen. PPL | H |
|---|---|---|---|---|---|---|---|---|---|
| Data (reference) | — | 9.67 | 3.96 | — | 53.51 | 4.31 | — | 14.15 | 5.44 |
| AR (sampling) | 128 | 8.89 | 4.01 | — | — | — | 1024 | 35.45 | 5.58 |
| MDLM | 128 | 17.89 | 4.03 | 128 | 115.45 | 4.33 | 1024 | 105.15 | 5.63 |
| DUO | 128 | 19.63 | 4.01 | 128 | 97.30 | 4.30 | 1024 | 77.69 | 5.55 |
| SEDD | — | — | — | — | — | — | 1024 | 101.43 | 5.61 |
| CANDI | 128 | 22.74 | 4.03 | 128 | 124.64 | 4.34 | 1024 | 143.13 | 5.71 |
| FLM | 128 | 57.50 | 4.17 | 128 | 112.33 | 4.33 | 1024 | 62.23 | 5.33 |
| FMLM+ | 128 | 15.43 | 3.90 | — | — | — | 128 | 66.60 | 5.21 |
| NCM | 12 | 15.41 | 3.98 | 12 | 93.05 | 4.31 | 32 | 83.12 | 5.42 |
Robustness check (Appendix E.1): rescoring LM1B samples with Mistral-7B, Llama-3.1-8B, Gemma-2-9B and Qwen-3-14B, NCM has the lowest average Gen. PPL (83.2 vs DUO 92.5) at 12 steps vs 128, with 13.4B vs 66B training tokens.
Conditional generation (paper Table 2)
| Setting | Method | Size | WMT14 BLEU | NFE | Size | XSum R-1 | R-2 | R-L | NFE |
|---|---|---|---|---|---|---|---|---|---|
| Reference | AR | 99M | 25.2 | 64 | 99M | 30.5 | 10.2 | 24.4 | 64 |
| No guidance | MDLM | 99M | 18.4 | 64 | 99M | 33.4 | 11.6 | 25.8 | 64 |
| DUO | 205M | 21.3 | 64 | 205M | 31.4 | 10.1 | 25.0 | 64 | |
| E2D2 | 99M | 24.8 | 64 | 99M | 28.4 | 8.3 | 22.0 | 64 | |
| ELF | 140M | 17.2 | 64 | 140M | 34.2 | 11.1 | 26.4 | 64 | |
| AURORA-LM | — | — | — | 130M | 34.7 | 12.2 | 26.6 | 128 | |
| NCM | 120M | 22.6 | 11 | 88M | 35.0 | 12.8 | 28.0 | 8 | |
| Guidance / MBR | ELF + CFG | 140M | 26.4 | 128 | 140M | 36.0 | 12.2 | 27.8 | 128 |
| AURORA-LM + CFG | — | — | — | 130M | 36.3 | 13.4 | 28.6 | 184 | |
| NCM + CFG | 120M | 23.9 | 22 | 88M | 35.3 | 13.0 | 28.2 | 16 | |
| NCM + MBR | 120M | 25.2 | 132 | 88M | 37.3 | 14.0 | 29.6 | 128 |
MBR settings: K=16 candidates on XSum (128 NFE), K=12 on WMT14 (132 NFE). CFG counted as 2 NFE per step. On translation NCM trails E2D2 unguided (22.6 vs 24.8) and ELF guided (25.2 vs 26.4), so it is best-in-class on summarization and second-tier on translation. †-marked baseline numbers come from Hu et al. (ELF).
Reasoning: Sudoku, % boards completely solved (paper Table 3; baselines from Agarwal et al.)
| Type | Method | Size | NFE | Easy (40 clues) | Medium (35) | Hard (30) |
|---|---|---|---|---|---|---|
| AR | Sample / Greedy | 25M | 128 | 13.9 / 14.6 | 5.1 / 5.1 | 0.6 / 1.0 |
| Discrete diff. | MDLM | 29M | 128 | 92.0 | 77.1 | 30.2 |
| DUO | 29M | 128 | 96.3 | 84.7 | 58.4 | |
| Continuous | CANDI | 29M | 128 | 79.3 | 45.9 | 16.7 |
| FLM | 29M | 128 | 94.2 | 82.7 | 44.5 | |
| S-FLM | 29M | 128 | 94.8 | 85.2 | 45.0 | |
| FMLM+ | 29M | 16 | 97.7 | 92.8 | 81.3 | |
| Ours | NCM | 25M | 12 | 99.5 | 96.5 | 86.1 |
Scaling (preliminary)
- Model size: 97M → 317M → 725M generators on LM1B with the same frozen tokenizer; Gen. PPL improves and entropy stays high at a fixed token budget.
- Test time: MBR over K candidates improves all metrics monotonically with diminishing returns (+4.48 ROUGE-1 from K=1 to K=256). At matched wall-clock, NCM+MBR beats diffusion baselines on XSum by ≈7% ROUGE-L.
- Training budget: 13.36B positions on LM1B (4.9× fewer than baselines' 66B); 127B on OpenWebText (≈4× fewer than MDLM/DUO's 500B+), tokenizer included.
Ablations that tell you what matters (Appendix F)
- More levels help (K = 4 → 32 on TinyStories), with diminishing returns; all reconstruct ≥99.95%.
- Attention pattern: block-diagonal 15.41 PPL / 5.65 s vs block-causal 19.63 / 7.90 s vs 2-level sliding window 17.84 / 6.83 s. Simplest is best and fastest.
- Corruption rate: moderate (~0.5 mean, linearly decreasing by level) greatly improves PPL; too much destroys the conditioning signal.
- Loss weight α: irrelevant for α ≤ 1.
- Level schedule: six shapes tested (linear, quadratic, sqrt, log, exp, cosine) at K=12.
Honest limitations (Appendix I, plus what the tables imply)
- Scale: main results at ~100M parameters; scaling study stops at 725M on LM1B. No billion-parameter or long-context evidence yet.
- Fixed hierarchy: K and every ℓk are fixed per dataset, so a 5-token reply still walks all K levels. Input-adaptive schedules are future work.
- Two-stage: tokenizer trained separately from the generator; representation and generation objectives are not aligned. One tokenizer and one generator per dataset.
- Reconstruction ceiling: 98.8–99.99% exact recovery. Errors are rare but are entity/number swaps, exactly the kind that matter.
- Artifacts at coarse-level transitions ("remain remain remain", "and and") show up in intermediate and sometimes final outputs.
- Not beating AR on quality at equal size; the win is quality-per-NFE and quality-per-second against parallel methods. Translation trails the best diffusion model.
- Fixed output length L per model (48, 64, 128, 1,024), inherited from the tokenizer; variable-length generation isn't addressed.
Cheat sheet
| Next-token (AR) | Masked / continuous diffusion | Block diffusion | Next-concept (NCM) | |
|---|---|---|---|---|
| Sequential steps for L tokens | L | T (64–1,024 in papers) | L / block × steps-per-block | K (8–32) |
| Positions per step | 1 (with KV cache) | L, every step | block size | ℓk: 1, 2, 4… up to L |
| Attention | causal | bidirectional | causal across blocks, bi within | block-diagonal (+ prefix for prompts) |
| KV cache | yes | no (not directly) | yes | not needed; early steps are tiny |
| Can revise earlier output? | no | yes | within block | yes, finer levels correct coarser ones |
| Training | 1 stage, 1 pass/token | 1 stage, many passes (500B+ tok on OWT) | 1 stage | 2 stages (tokenizer → generator), ~4–5× fewer tokens |
| Few-step quality | n/a | collapses without distillation | moderate | designed for few steps |
| Output length | variable (EOS) | fixed canvas | variable | fixed L per model |
| Best for | quality per parameter, chat | global edits, infilling | serving speed with cache | latency-bound generation, planning tasks (Sudoku), cheap MBR |
| Examples | GPT, Llama, Qwen | MDLM, SEDD, DUO, LLaDA, Mercury, DiffusionGemma | BD3-LM, Fast-dLLM v2, DFlash | NCM (this paper) |
When would you reach for NCM today?
- You care about latency for fixed-length outputs (headlines, translations, structured fills) and can train per-task models at ~100M scale.
- The task needs global consistency (constraint satisfaction, layouts, structured outputs): coarse-first generation is a natural planner.
- You want cheap best-of-N / MBR: 16 NCM candidates cost what one diffusion sample costs.
When you wouldn't (yet)
- Open-ended chat with variable length and billions of parameters: no evidence yet, and a frozen per-dataset tokenizer is a real operational burden.
- You need a shared vocabulary/tokenizer across tasks or languages (WMT14's tokenizer already shows utilization dropping to 62%).
Try it
Released: tokenizer + generator checkpoints and tokenized test sets for XSum, WMT14, Sudoku (easy/medium/hard), LM1B (small/medium/large), TinyStories, OpenWebText. MIT license. Repo · HF collection.
Bonus · The family tree, so you can keep going
Where the pieces come from
- VQ-VAE (2017)
- Discrete latent codes + codebook + straight-through estimator. The ancestor of every "tokenizer for continuous data".
- VQ-VAE-2 / VQGAN
- Hierarchical and adversarially-trained discrete image tokenizers.
- SoundStream / EnCodec
- Residual vector quantization for audio: quantize, take the error, quantize again.
- RQ-VAE / RQ-Transformer
- Residual quantization for autoregressive image generation.
- VAR (2024)
- Next-scale prediction: generate an image coarse-to-fine, one resolution at a time, every token of a scale in parallel. NCM is VAR for text. HMAR (CVPR 2025, same first author as NCM) is the hierarchical masked variant.
- ImageFolder
- Source of the nested-dropout trick NCM uses on its hierarchy.
The competition: diffusion & flow language models
- D3PM → SEDD → MDLM
- The discrete (masked) diffusion lineage; MDLM is the standard ~100M baseline.
- DUO (The Diffusion Duality)
- Links discrete and Gaussian diffusion; strongest unconditional baseline on LM1B.
- LLaDA, Mercury, DiffusionGemma
- Diffusion LMs at billions of parameters, including commercial serving.
- Block Diffusion (BD3-LM), Fast-dLLM v2, DFlash
- Hybrids that regain the KV cache by generating blocks left-to-right; DFlash uses diffusion as a speculative drafter for AR models.
- CDCD, CANDI, ELF, AURORA-LM, Flow-map LMs
- Continuous / flow-based text generation in embedding or latent space; AURORA-LM and ELF are the strongest conditional baselines in the paper.
- Eso-LM, Posterior refinement (FMLM+)
- Any-order models; FMLM+ is the strongest Sudoku baseline (81.3% hard at 16 NFE).
Cousins: hierarchical and "concept-level" LMs
- MEGABYTE, H-Net
- Hierarchies over bytes/chunks for efficiency; still left-to-right.
- Large Concept Models, Dynamic LCM
- Predict sentence-level embeddings instead of tokens.
- ConceptLM (2026)
- "Next concept prediction in discrete latent space": discrete concepts that guide subsequent token generation. Closest name-neighbour to NCM, different mechanism (concepts then tokens, not a residual hierarchy of the whole sequence).
- HDLM (NeurIPS 2026)
- Hierarchical diffusion over semantic vocabularies: coarse-to-fine, but via diffusion rather than one parallel pass per level.
Natural next reads
- Serving systems for parallel decoders: PagedAttention/vLLM and Efficiently Scaling Transformer Inference: NCM's claim that inference is "the dominant workload" rests on these.
- Sampling-time compute: It's MBR all the way down.
- The authors' own "what's next": joint tokenizer+generator training, input-adaptive depth, and scaling beyond 725M. Watch the repo.
Verdict · What is NCM, and can it replace LLMs and diffusion models?
In three sentences. NCM is a two-stage text generator: a multiscale residual VQ-VAE turns a fixed-length text into a pyramid of discrete codes (a few coarse codes on top, the full sequence at the bottom), and a standard transformer generates that pyramid one level at a time, every code in a level in parallel. It is the strongest parallel text generator published at the ~100M scale: 10–31× faster than the best diffusion models at equal or better quality, with 4–5× less training data and no distillation. It is not, and does not claim to be, a replacement for today's LLMs.
Scorecard, October 2026
| Criterion | Autoregressive LLM | Diffusion LM | NCM |
|---|---|---|---|
| Quality at equal size | best (25.2 BLEU, lowest PPL) | behind AR; strongest on some conditional tasks | beats diffusion on 5 of 6 tasks; trails AR on translation and PPL |
| Latency (sequential passes) | L = 128–1,024 | T = 64–1,024, each full-width | K = 8–32, early passes tiny |
| Can revise earlier output | no | yes | yes, by design (residual levels) |
| Variable-length output | yes (EOS token) | fixed canvas, padding tricks | fixed L per model; no mechanism yet |
| One tokenizer for all text | yes | yes (same BPE) | one frozen VQ-VAE per dataset; 62–90% utilization |
| Lossless representation | yes | yes | 98.8–99.99%; errors are names, numbers, markup |
| Scale proven | trillions of tokens, 1012 params | billions of params, in production (Mercury 2, DiffusionGemma) | ≤725M params, one dataset |
| Long context / multi-turn | yes | limited | prefix ≤512 tokens; untested |
| Reasoning via long CoT + RL | the current frontier recipe | early work | no story; Sudoku shows planning, not deliberation |
| Training data efficiency | 1 pass/token | 500B+ tokens on OWT, many epochs | 4–5× fewer token exposures than diffusion (FLOPs unreported) |
| Cheap test-time scaling (MBR, best-of-N) | costly | very costly | 16 candidates for the price of one diffusion sample |
| Production readiness | everywhere | hosted APIs exist | eval-only code, CUDA-only, no serving stack |
Green = strong, amber = partial, red = blocker. Every NCM cell traces to the paper (Tables 1–3, 5–8, Appendix I) or the repo; diffusion production status from benchlm's diffusion-LLM evidence table.
So, can it replace LLMs? No. Six hard blockers
- Fixed output length. The finest pyramid level is the tokenizer's output grid: 48, 64, 128 or 1,024 tokens, always. There is no end-of-sequence token. Chat, code and agent loops need variable length. Inherited from VAR, where images have a fixed resolution and text doesn't. (Paper App. I)
- One frozen tokenizer per dataset. An LLM has one vocabulary for everything. NCM's WMT14 tokenizer cannot encode XSum, and as the text domain widens the tokenizer visibly strains: WMT14 (151k-word vocabulary) drops to 61.9% codebook utilization and 98.79% reconstruction. Nobody has trained a general-text multiscale VQ-VAE. (Tables 1, 5)
- Lossy representation. 0.01–1.2% of tokens come back wrong, and they are named entities, numerals, diacritics, markup: exactly the tokens that matter for facts and code. An LLM has no such floor. (Table 1, App. G)
- No evidence beyond 725M parameters or a 512-token prefix. No long context, no multi-turn, no instruction following. (§4.5, Table 6)
- Never timed against an AR model. All speedups are vs diffusion. A same-size AR model still wins translation (25.2 vs 22.6 BLEU) and perplexity; at batch 16 a 100M AR model with a KV cache is also fast. (Tables 2, 8)
- No reasoning story. Frontier progress since 2024 is long, variable-length chain-of-thought trained with RL. NCM's Sudoku win is constraint satisfaction over a fixed board, a real but different capability.
Can it replace diffusion LMs? Not yet, but this is the real contest
Diffusion LMs already ship at billions of parameters with hosted APIs (Mercury 2 ≈ 800 tok/s, DiffusionGemma >1,000 tok/s on one H100) and are patching their weaknesses with block diffusion (KV cache back) and speculative drafting (DFlash). NCM beats them cleanly at 100M on quality-per-pass and wall-clock, and its data efficiency is a bigger lever than its speed. But it has to be re-validated at 1–8B with a general tokenizer before it is a competitor rather than a paper. The authors' own "What's next" (scale, joint tokenizer+generator training, adaptive depth) is exactly that list.
What is stopping us from training NCM models today?
Not access: MIT code, HF checkpoints, a GPT-2-shaped transformer, standard PyTorch. What stops it is research risk:
- Does a universal multiscale text tokenizer exist? Images downsample naturally; text does not. This is the open scientific question, and everything else depends on it.
- Variable length needs a new mechanism: a length code at level 1, padding codes, or a data-dependent K. Unsolved.
- Joint training of tokenizer and generator would remove the two-stage brittleness, but VQ codebooks are notoriously unstable to train end-to-end.
- Real cost accounting. The 4–5× saving counts token exposures, not FLOPs, and the tokenizer costs about as much as the generator. Nobody has published a FLOP-matched comparison.
- Incentives. The labs with the compute have years of AR + RL infrastructure. NCM needs a champion willing to spend ~1B-parameter-scale compute on a representation bet, and needs independent replication first (all results so far are author-reported).
How it could steer AI progress
- Separating "what to say" from "how to say it". The coarse levels are a learned, discrete plan. That is a natural place to search, verify, or run RL, on plans instead of tokens, and makes test-time compute cheap and structured.
- Inference economics. K ≈ 12–32 weight reads per output instead of 128–1,024 moves small-batch decoding from memory-bandwidth-bound toward compute-bound. A different hardware target than today's HBM race; good news for local inference.
- Hybrids are the likely winner. AR for variable-length deliberation, pyramid decoding for the "render" step; or NCM as a drafter inside speculative decoding. The lineage VAR → HMAR → NCM points at masked/hierarchical hybrids next.
- Data efficiency. If the 4–5× token saving survives scale and FLOP accounting, that is the most valuable result in the paper, more than the 31×.
The one question to watch, and a 2027 checklist
Re-evaluate when you see any of these:
- An NCM-style model ≥1B parameters on a general web corpus with a single tokenizer.
- A variable-length mechanism with results on open-ended generation.
- A FLOP-matched (not token-matched) comparison against AR and diffusion, including AR wall-clock.
- Independent replication of the LM1B / XSum numbers.
- An instruction-tuned or RL-tuned NCM, or NCM used as a speculative drafter in a production AR stack.
Sources: NCM paper · Hazy Research blog · AI Angst: why it isn't the end of LLMs · benchlm: diffusion LLMs ranked by evidence · DiffusionGemma explained · VAR · Kai, "NCM: Is this the End of LLMs?". The scorecard colours and the "what will labs do" paragraphs are editorial judgement; the cell contents are sourced.