Three live threads (2024 → mid-2026)

1. Is the tokenizer removable? Byte Latent Transformer (2024) 2412.09871 learns dynamic byte patches and matches token-based models at scale. Whether byte-level models displace BPE or remain a research direction is open.

2. Data is now the binding constraint. FineWeb 2406.17557 and DCLM 2406.11794 show curation beating scale, while Muennighoff et al. 2305.16264 studies what happens when you run out and must repeat data. Lee et al. 2107.06499 established deduplication as mandatory — though FineWeb later found more aggressive global dedup can hurt, so even this is not simple.

3. Does RL add capability? DeepSeek-R1 2501.12948 made reasoning RL mainstream; Yue et al. 2504.13837 argues it sharpens rather than extends. Unresolved, and the most consequential open question on this page.

← Scale, systems, and alignment The complete reference list →