The 6ND rule

Faded derivation: why training costs ≈ 6 FLOPs per parameter per token

1Count the forward cost of one weight in one matrix multiply, per token.

1 multiply + 1 add = 2 FLOPs

Every weight participates in exactly one multiply-accumulate per token, so the forward pass costs 2N FLOPs per token for N parameters.

2The backward pass computes two different gradients. Name them.

The gradient with respect to the layer’s inputs (to keep propagating backwards) and with respect to its weights (to update them). Each is a matrix product of the same shape class as the forward one.

3Total the three passes.

2N forward + 2N grad wrt input + 2N grad wrt weight = 6N per token

Over D tokens the run costs C ≈ 6ND FLOPs. Backward is twice forward, which is the fact worth remembering.

4What does this expression ignore, and when does that bite?

The attention score computation, which has no parameters and so contributes nothing to N, yet costs 4Ld per layer per token. From the previous part that stays a minority until L > 6d. 6ND is an excellent approximation at ordinary context lengths and an increasingly bad one at very long ones.

← AdamW, and why the W matters Splitting the budget: Chinchilla →