The modern block

RMSNorm 1910.07467 showed re-centering was never load-bearing; RoPE 2104.09864 made relative position fall out of an absolute rotation; Shazeer (2020) 2002.05202 found the gated feed-forward variant that every current model uses, and with it the 8/3 width convention. On the cache side, MQA 1911.02150 and GQA 2305.13245 traded quality against KV-cache size, and Shazeer et al. (2017) 1701.06538 then Switch 2101.03961 made conditional computation work at scale.

← The foundations Scale, systems, and alignment →