Part 9 · The Papers & Sources
The complete reference list
| work | authors · identifier | why it is here |
|---|---|---|
| Attention Is All You Need | Ashish Vaswani et al., 2017 · arXiv:1706.03762 | The transformer. Removes recurrence; source of the scaled dot-product formula. |
| Language Models are Few-Shot Learners | Tom B. Brown et al., 2020 · arXiv:2005.14165 | GPT-3: in-context learning emerges from scale alone. |
| Neural Machine Translation of Rare Words with Subword Units | Rico Sennrich et al., 2015 · arXiv:1508.07909 | Brings BPE into NLP; ends the unknown-word problem. |
| SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing | Taku Kudo et al., 2018 · arXiv:1808.06226 | Language-agnostic, reversible tokenization. |
| Byte Latent Transformer: Patches Scale Better Than Tokens | Artidoro Pagnoni et al., 2024 · arXiv:2412.09871 | Dynamic byte patches; the strongest tokenizer-free result. |
| Root Mean Square Layer Normalization | Biao Zhang et al., 2019 · arXiv:1910.07467 | Drops mean subtraction; cheaper and equally effective. |
| RoFormer: Enhanced Transformer with Rotary Position Embedding | Jianlin Su et al., 2021 · arXiv:2104.09864 | Relative position from absolute rotation. |
| GLU Variants Improve Transformer | Noam Shazeer et al., 2020 · arXiv:2002.05202 | SwiGLU, and the 8/3 hidden-width convention. |
| Fast Transformer Decoding: One Write-Head is All You Need | Noam Shazeer et al., 2019 · arXiv:1911.02150 | MQA: one KV head for the whole layer. |
| GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints | Joshua Ainslie et al., 2023 · arXiv:2305.13245 | Interpolates MHA and MQA; the current default. |
| Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer | Noam Shazeer et al., 2017 · arXiv:1701.06538 | Conditional computation: parameters without proportional FLOPs. |
| Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity | William Fedus et al., 2021 · arXiv:2101.03961 | Top-1 routing makes MoE simple and stable. |
| DeepSeek-V3 Technical Report | DeepSeek-AI et al., 2024 · arXiv:2412.19437 | MLA plus auxiliary-loss-free balancing at 671B/37B. |
| Scaling Laws for Neural Language Models | Jared Kaplan et al., 2020 · arXiv:2001.08361 | Loss follows smooth power laws in N, D and C. |
| Training Compute-Optimal Large Language Models | Jordan Hoffmann et al., 2022 · arXiv:2203.15556 | Chinchilla: scale parameters and tokens together. |
| Chinchilla Scaling: A replication attempt | Tamay Besiroglu et al., 2024 · arXiv:2404.10102 | Shows the published Approach-3 constants are internally inconsistent. |
| Scaling Data-Constrained Language Models | Niklas Muennighoff et al., 2023 · arXiv:2305.16264 | What to do when the data, not the compute, runs out. |
| Decoupled Weight Decay Regularization | Ilya Loshchilov et al., 2017 · arXiv:1711.05101 | AdamW: decay the weights, not the gradient. |
| Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer | Greg Yang et al., 2022 · arXiv:2203.03466 | muP: tune small, transfer to large. |
| FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness | Tri Dao et al., 2022 · arXiv:2205.14135 | Exact attention, restructured around memory traffic. |
| FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning | Tri Dao et al., 2023 · arXiv:2307.08691 | Better work partitioning; roughly doubles throughput. |
| Online normalizer calculation for softmax | Maxim Milakov et al., 2018 · arXiv:1805.02867 | The streaming-softmax trick FlashAttention is built on. |
| Training Deep Nets with Sublinear Memory Cost | Tianqi Chen et al., 2016 · arXiv:1604.06174 | Gradient checkpointing: √n memory for one extra forward pass. |
| ZeRO: Memory Optimizations Toward Training Trillion Parameter Models | Samyam Rajbhandari et al., 2019 · arXiv:1910.02054 | Shard optimizer state, gradients, then parameters. |
| Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | Mohammad Shoeybi et al., 2019 · arXiv:1909.08053 | Tensor parallelism inside the layer. |
| GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism | Yanping Huang et al., 2018 · arXiv:1811.06965 | Pipeline parallelism with micro-batches. |
| Ring Attention with Blockwise Transformers for Near-Infinite Context | Hao Liu et al., 2023 · arXiv:2310.01889 | Context parallelism with communication hidden behind compute. |
| Training language models to follow instructions with human feedback | Long Ouyang et al., 2022 · arXiv:2203.02155 | InstructGPT: the RLHF pipeline that made models usable. |
| Direct Preference Optimization: Your Language Model is Secretly a Reward Model | Rafael Rafailov et al., 2023 · arXiv:2305.18290 | Closed-form solution removes the reward model and the RL loop. |
| DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models | Zhihong Shao et al., 2024 · arXiv:2402.03300 | GRPO: the group replaces the value network. |
| DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning | DeepSeek-AI et al., 2025 · arXiv:2501.12948 | Reasoning RL with verifiable rewards, at scale. |
| Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? | Yang Yue et al., 2025 · arXiv:2504.13837 | Argues RLVR sharpens sampling rather than adding capability. |
| The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale | Guilherme Penedo et al., 2024 · arXiv:2406.17557 | Open, documented web curation at 15T tokens. |
| DataComp-LM: In search of the next generation of training sets for language models | Jeffrey Li et al., 2024 · arXiv:2406.11794 | Controlled benchmark showing curation beats raw scale. |
| Deduplicating Training Data Makes Language Models Better | Katherine Lee et al., 2021 · arXiv:2107.06499 | Deduplication as a first-class training-data operation. |
| The Pile: An 800GB Dataset of Diverse Text for Language Modeling | Leo Gao et al., 2020 · arXiv:2101.00027 | The first widely-used open diverse pretraining corpus. |
| Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer | Colin Raffel et al., 2019 · arXiv:1910.10683 | T5 and C4; the cleaned-Common-Crawl template. |
| DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining | Sang Michael Xie et al., 2023 · arXiv:2305.10429 | Learns domain weights instead of guessing them. |