Mixture of experts: parameters without proportional FLOPs

Replace the single feed-forward network with E of them plus a router. The router scores experts, keeps the top k, and mixes their outputs:

y = ∑_{i∈TopK(g(x))} g_i(x) · FFN_i(x), g(x) = softmax(xW_r)

E the number of experts · k how many are used per token · W_r the router's projection · g_i(x) the gate weight expert i receives

Total parameters scale with E; FLOPs per token scale with k. DeepSeek-V3 2412.19437 carries 671B parameters but activates 37B per token — an 18× gap between what the model knows and what any single token pays for.

The failure mode is routing collapse: nothing in the loss stops the router from sending everything to one expert. The classical fix adds an auxiliary load-balancing loss, which fights the language-modelling objective. The newer approach adds a per-expert bias to the routing scores and adjusts it based on observed load, balancing without touching the gradient of the main loss at all.

← The Maths IV: Inside the SOTA Methods Shrinking the KV cache: MQA, GQA, MLA →