Study: Data Improvements Outpace Model Gains in Pretraining Efficiency from 2019 to 2025

ChainCatcher report: On September 8, 2026, Dwarkesh Patel and Jerry Han published a study dissecting the contributions of data and model improvements to pretraining progress from 2019 to 2025. The study was conducted at a smaller scale focused on pretraining, using annually published model architectures and publicly available corpora, with training computed up to a maximum of 1e19 FLOPs. Results show that under a 1e19 FLOPs computational budget, data improvements yielded a 12.0x gain in computational efficiency, while model improvements yielded a 3.7x gain—meaning data-side gains were approximately 3.24 times larger than model-side gains. The gains from data and model improvements were largely independent and exhibited minimal interaction; under a linear model, their additive effect explained 88% of the variance in OLMES scores. Model-side advancements evolved from GPT-2 to OLMo-2, encompassing improvements in optimizers, positional encodings, normalization, activation functions, and initialization. Data-side advancements progressed from OpenWebText (~9 billion tokens in 2019) to larger, more finely filtered corpora such as UltraFineWeb by 2025.