Study: Data Improvements Outpace Model Gains in Pre-training Efficiency from 2019 to 2025

Huo Xing Cai Jing reports that on September 8, 2026, Dwarkesh Patel and Jerry Han released a study dissecting the contributions of data and model improvements to pretraining advancements from 2019 to 2025. The study was conducted at a smaller scale focused specifically on pretraining, using publicly available model recipes and training corpora for each year, and combining training up to a maximum computational scale of 1e19 FLOPs. Results showed that under a 1e19 FLOPs computational budget, data improvements yielded a 12.0x gain in computational efficiency, while model improvements yielded a 3.7x gain—meaning data-side gains were approximately 3.24 times larger than model-side gains. The gains from data and model improvements were largely independent and exhibited minimal interaction; under a linear model, their additive effect explained 88% of the variance in OLMES scores. Model-side advancements evolved from GPT-2 to OLMo-2, encompassing improvements in optimizers, positional encoding, normalization, activation functions, and initialization. Data-side advancements progressed from OpenWebText, containing approximately 9 billion tokens in 2019, to larger, more finely filtered corpora such as UltraFineWeb by 2025.