Counting what it costs

Per layer, with hidden size d and feed-forward size d_ff, the forward pass costs (in FLOPs per token, counting a multiply-add as 2):

6d² (QKV) + 2d² (output) + 4Ld (scores and values) + 2·n_mat·d·d_ff (FFN)

d the model width · d_ff the feed-forward hidden width · L the context length · n_mat the number of FFN matrices (2 classic, 3 for SwiGLU)

Set d_ff = 4d with two matrices and the quadratic term overtakes everything else only when 4Ld > 24d², that is L > 6d.

Attention’s share of block FLOPs, computed exactly from the expression above. For a 4096-wide model the crossover is at about 24,000 tokens of context. Below that, the feed-forward network dominates and attention is not the bottleneck — which is why the first wave of efficient-attention work often failed to speed anything up in practice.

← The rest of the block: RMSNorm and SwiGLU Quiz: The Maths II: Inside the Transformer →