Shrinking the KV cache: MQA, GQA, MLA

During generation every past key and value must be kept. For multi-head attention that is

2 · n_layers · n_heads · d_head · L · 2 bytes

the leading 2 counts keys and values · n_layers, n_heads, d_head the model's depth, head count and head width · L the tokens cached so far · the trailing 2 bytes is bf16

which grows linearly with context and quickly exceeds the weights themselves. MQA 1911.02150 shares one key/value head across all query heads — a large saving and a real quality loss. GQA 2305.13245 interpolates: g groups of query heads share a key/value head, recovering most of the quality at a fraction of the cache. Multi-head latent attention instead caches a low-rank latent that keys and values are reconstructed from, decoupling cache size from head count entirely.

The same three variants across four orders of magnitude of context. The gap is not a constant factor to be engineered away later — it decides whether long-context serving is possible on a given machine at all.

← Mixture of experts: parameters without proportional FLOPs DPO: preference learning without the RL →