Part 2 · Intuition

Attention as soft lookup

Recurrence processes a sequence by carrying a hidden state forward one step at a time. Two things go wrong: information from far back has to survive many overwrites, and step t cannot be computed until step t−1 is done, so a GPU with tens of thousands of cores runs one position at a time.

Attention replaces the carried state with direct access. Every position emits a query (“what am I looking for?”), every position emits a key (“what do I offer?”) and a value (“what I will hand over”). Each query is compared against all keys at once, the comparisons are softmaxed into weights, and the output is the weighted average of values. Every position is computed independently, so the whole sequence goes through in one parallel pass.

The name to keep: attention trades memory for parallelism. It removes the sequential dependency, but it must materialize a comparison between every pair of positions, and that is quadratic. Part 5 shows exactly when that quadratic term starts to hurt — the answer is later than most people assume.