Why the 1/√dk is not cosmetic

The scaling factor is the single most-copied and least-explained line in the architecture. It has a short derivation, and the derivation predicts exactly what the widget below shows.

Faded derivation: the temperature of a dot product

1Model q and k as independent vectors with zero-mean, unit-variance components. What is E[q·k]?

𝔼[q·k] = ∑_i 𝔼[q_i]𝔼[k_i] = 0

Independence lets the expectation factor; each factor is zero.

2Now the variance. This is the step that matters.

Var(q·k) = ∑_i Var(q_ik_i) = ∑_i 1 = d_k

Variances of independent terms add. So the logits have standard deviation √d_k — they grow as the head gets wider.

3What does a softmax do to inputs whose spread keeps growing?

It saturates. Softmax is scale-sensitive: multiply all logits by c > 1 and the distribution sharpens; as c → ∞ it becomes one-hot. With spread √d_k, a d_k = 128 head has logits several times larger than a d_k = 8 head, purely from dimensionality.

4Why is saturation fatal for training specifically?

∂softmax_i/∂z_j = p_i(δ_ij − p_j)

p the softmax output · z its input logits · δ_ij the Kronecker delta, 1 when i = j and 0 otherwise

When p is one-hot, every entry of that Jacobian is ~0: p_i ≈ 0 kills most terms and p_i ≈ 1 gives p(1−p) ≈ 0. No gradient flows back through a saturated softmax, so the attention pattern cannot be learned.

5Choose the divisor that makes the logit variance independent of width.

Var( q·k√d_k ) = d_kd_k = 1

Dividing by √d_k holds the entering variance at 1 for every head width, so the softmax operates in the same regime whether the head is 8 or 128 wide. This is exactly the justification given in the original paper.

Step 2 is the one to remember: variances add, so dot products grow like √d. The same argument governs initialization schemes and is worth having in reflex memory.

This is not a reconstruction after the fact. The original paper gives the same argument in a footnote, suspecting that “for large values of d_k, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients” — steps 2 through 4 above, in one sentence.

← Attention, derived Position without recurrence: RoPE →