Part 4 · The Maths II: Inside the Transformer
Attention, derived
Each position emits three vectors by linear projection: a query q = xWQ, a key k = xWK, and a value v = xWV. The compatibility of query i with key j is their dot product; softmax over j turns compatibilities into weights that sum to one; the output is the weighted average of values.
Q, K, V the stacked query, key and value matrices, one row per position · dk the head dimension (the length of a single query or key vector) · M the causal mask
M is the causal mask: 0 where j ≤ i and −∞ above the diagonal, so position i cannot read the future. The mask must be added before the softmax; zeroing weights afterwards would leave the rows unnormalized.
Why this matters. Every dependency in the sequence is now one hop away and every position is computed in parallel. The price is the T×T matrix inside the softmax — the only term in the whole architecture that is quadratic in sequence length.