Autoregressive generation: the typewriter

Almost every LLM you have used — GPT, Claude, Gemini, Llama — generates text autoregressively: it predicts one token, appends it, feeds the extended text back in, predicts the next, and so on, strictly left to right. Mathematically it factorizes the probability of a sentence as a chain of next-token predictions:

P(x) = P(x₁) · P(x₂ | x₁) · P(x₃ | x₁, x₂) · ⋯ · P(x_N | x₁, …, x_N−1)

x a sequence of N tokens · x_i the token at position i · each factor conditions only on the tokens before it — the autoregressive choice

Note something important: this factorization is fully general. The chain rule of probability lets you decompose any joint distribution this way. Autoregression’s limits are not about what it can represent — they are about how it must compute.

The

→

cat

→

sat

→

the

→

mat

The typewriter: six tokens means six full forward passes through the network, each one waiting for the previous to finish. Nothing already typed can ever be changed.

Two properties of this design matter for our story:

It is inherently sequential. Token 500 cannot be computed until token 499 exists — its sampled value is an input to the next pass. (During training you can evaluate all positions in parallel because the true text is known; at inference the inputs don’t exist until you sample them.) Generating N tokens costs N forward passes, one after another. Worse, each decode step is memory-bandwidth-bound: the GPU spends its time streaming billions of weights and the KV cache from memory to compute a single token’s worth of arithmetic. The chip is starved for data, not arithmetic.
It is write-once. Once a token is emitted it is frozen. If the model commits to a word that turns out to be a poor fit three words later, it cannot go back — it can only try to recover in the text that follows.

For a chatbot answering a single question, this was tolerable. But the field is moving toward agents and reasoning models that generate thousands of tokens of intermediate thinking, call tools, and chain many steps together. Every extra token of “thinking” is paid for in wall-clock time, multiplied across every step of a workflow. (Note the one thing faster decoding cannot fix: agent round k+1 still has to wait for round k — which is precisely why the per-round latency is the lever that matters.) Inception’s framing on the very page we’re studying: “AI shouldn’t behave like a one-way typewriter” — it should behave like an editor.

Here is the whole argument as one picture — hover to compare (the y-axis is logarithmic):

An AR model’s sequential passes grow linearly with output length; a dLLM’s stay flat at its step count. At N = 1,000 tokens the gap is 1,000 passes versus a few dozen — and this ratio, not raw FLOPs, is what sets wall-clock latency in the bandwidth-bound decode regime described above.

← Deep background: what a language model does Meanwhile, in image generation: diffusion →