Data improvements outperform model advances in AI pre-training efficiency

Author: Dwarkesh Patel

Compiled by Deep潮 TechFlow

Shenchao Summary: Over the past six years, AI model capabilities have advanced rapidly, but has this progress come primarily from algorithms or data? This article presents a counterintuitive conclusion through a series of small-scale controlled experiments: improvements in data have boosted computational efficiency more than three times as much as improvements in models. For those focused on AI infrastructure investment and competition among leading labs, it reveals an underappreciated driver: data engineering.

Over the past few years, how much of the rapid progress in AI has come from improvements in data versus improvements in models? The answer to this question has significant implications for the economic models of leading laboratories and the pace of future advancements.

We conducted a relatively small-scale investigation focused specifically on pre-training from 2019 to 2025. During these years, a new set of open-source model architectures was published annually, summarizing publicly known algorithmic advancements of that year—such as improvements in architecture, optimizers, initialization, learning rate scheduling, and hyperparameters. Simultaneously, new publicly available corpora emerged each year, generated through larger-scale web crawls and novel curation, extraction, and filtering techniques.

We trained various combinations of model architectures and data corpora representing different year-level benchmarks, across varying scales of training compute (up to 1e19 FLOPs).

Clearly, we cannot compare these different models based on their cross-entropy loss on a fixed dataset, since we are changing the datasets on which they are trained. Instead, we evaluate these models based on their final capabilities, using the OLMES evaluation (which aggregates 10 relatively simple benchmarks, mostly multiple-choice question answering). Unfortunately, evaluating final capability rather than pretraining loss introduces some noise into our results, as you will see in the figures below; however, we attempt to obtain cleaner bounds by using multiple random seeds.

We found that, between 2019 and 2025, under a computational budget of 1e19 FLOPs, more than 3.24 times the improvement in computational efficiency came from data improvements rather than model improvements (data: 12.0x, model: 3.7x).

This grid shows the improvement in final test performance of our models, compared to the 2019 data and architecture baseline, under 3.16e18 FLOPs of compute.

We found that the benefits from data improvements and model improvements are largely independent of each other, with no interaction effects (i.e., the benefit of a particular model improvement does not depend on a specific training dataset, and vice versa). Using a linear model, 88% of the variance in the OLMES score can be explained by the additive effects of model and data improvements.

As context, let’s briefly summarize the changes that occurred on the data side and the model side from 2019 to 2025.

On the model side, we have progressed from GPT-2 to OLMo-2, incorporating key innovations in optimizers, positional encoding, normalization, activation functions, and initialization.

On the data side, we began in 2019 with OpenWebText, which consisted solely of web pages linked to on Reddit and sufficiently upvoted; after deduplication and filtering, it ultimately contained only about 9 billion tokens—essentially the training data for GPT-2. By 2025, open-source datasets like UltraFineWeb are not only much larger in scale (by crawling the entire internet) but also employ far more sophisticated filtering methods—for example, training a classifier to predict which data actually improves model performance.

A naive interpretation of our results is that most of AI’s progress between 2019 and 2024 (the pretraining era) was due to better data engineering (extraction, curation, etc.), while the model work during that period was far less significant.

But this may be the wrong way to view the value of model improvements. The primary contribution of model improvements is not necessarily computational efficiency—achieving the same performance with fewer FLOPs. Instead, it first makes larger-scale computation accessible. As parameter counts, context lengths, runtime durations, and cluster sizes increase, various issues tend to fail—such as gradient explosion or vanishing, memory and bandwidth exhaustion, and training becoming impossibly slow. A significant portion of model research focuses on removing or delaying these scaling bottlenecks. Many of the most important innovations fall into this category, such as MoE, sparse attention variants, stability improvements (normalization placement, initialization, etc.), and system- and kernel-level optimizations like FlashAttention.

The data improvements we study here may be less significant for larger models. Small models—like the ones we train—benefit substantially from improved data quality, because their limited capacity requires careful selection of what to include. In contrast, large models have substantial excess capacity, so you might simply want to feed them as much data as possible, even if much of it is noise; the magic of stochastic gradient descent will still separate the signal from the noise. If you opt for aggressive filtering, you may need to run dozens of epochs, and empirically, this often yields worse results than using a larger dataset with slightly lower average quality. In fact, when considering that state-of-the-art models are often overtrained up to 100 times beyond the Chinchilla optimal standard to minimize inference compute for reinforcement learning and deployment, the risks of aggressive data curation become even greater.

An analogy might be the difference between a sailboat and a container ship: the container ship isn’t necessarily faster, but it can carry thousands of tons of cargo (equivalent to hundreds of trillions of tokens of pretraining data) and won’t capsize in rough seas (equivalent to stable training across hundreds of thousands of GPUs).

With today’s larger, more robust container ships, we no longer have to worry about what cargo to load—we can simply fill them with anything that’s even slightly useful. But for the small, fragile sailboats of 2019, you had to be extremely careful and carry only the most valuable goods.

But if the essence of progress in pretraining is merely loading more cargo onto this ship, are we running out of cargo? This is the issue of the data wall—and the extent to which synthetic data can help us overcome it. Synthetic data is already widely used in labs, yet we have barely studied whether it can effectively expand the data corpus without compromising model performance. If such gains are limited, the main driver of pretraining progress will stall, as we won’t generate more internet content, and the extent to which a fixed dataset can be curated is also finite. It should be noted that we currently have no positive evidence to suggest this will indeed happen. But given how crucial data appears to be in driving pretraining progress, this seems like a critical question worthy of further investigation.

Ryan Greenblatt noted that many historical improvements to pretraining data corpora appear to be the type of progress that automated researchers could directly drive through empirical testing—for example, running ablation experiments with models trained on different data to observe model performance. Thus, it aligns perfectly with our findings: if AI R&D becomes automated, the data-driven advances that have propelled pretraining progress since 2019 could accelerate dramatically.

We want to clarify that whether pretraining progress accelerates or decelerates in isolation is not the most important question regarding overall AI progress, as much of the gains over the past two years have come from reinforcement learning.

Future research directions

Here are some future research directions and questions that we believe would be both exciting and important:

• You can scale up this experiment to see whether the data or model improvements rely more on scale, thereby having a greater impact in cutting-edge areas.
• What is the marginal value of high-quality new data in pre-training and post-training, measured by final performance?
• We want to get a general sense of the real-world effectiveness of synthetic data. A specific question worth investigating is: If you have a small set of high-quality data, how much better is the performance when augmenting it with synthetic data generation, compared to simply training multiple rounds on the original data?
• You can estimate the implied value of data by comparing expenditures on data brokers, environmental producers, and similar areas relative to spending on computing power and researchers.

We want to explore the role that data plays in driving advancements in AI. There are many other ways to investigate this question, some of which may be more sophisticated or informative than our approach. Additionally, our experiment was very small-scale. We certainly suspect we may have missed something and would love to hear how others might approach this problem—ideally, we’d also love to see their results!

Special thanks to Charlie O'Neill for many insightful discussions.

We pre-trained these model recipes from scratch on diverse datasets using varying compute budgets and multiple independent seeds. Our compute budgets were: 1e17, 3.16e17, 1e18, 3.16e18, and 1e19 FLOP. Compute is calculated using the standard convention C = 6ND, where N is the number of non-embedding parameters and D is the number of data tokens.

For each compute budget, we adjust the number of parameters to vary the number of training tokens, determining the optimal compute allocation for each training recipe and corpus combination. We identify this optimal compute point using held-out loss on the corpus. From this, we derive the compute scaling curves for downstream performance of each combination and ultimately extract the compute multiplier.

We enforce the use of a shared tokenizer and context length across all runs: GPT-2 BPE (tiktoken, vocabulary size of 50,257) and T=2048, batch = 262,144 tokens.

The final performance of our training runs depends heavily on hyperparameters. Clearly, it's impossible to exhaustively test all possible hyperparameter combinations—hyperparameter tuning is indeed a refined art! We strive to maintain control over this process and treat the peak learning rate as the most critical hyperparameter.

Some algorithm versions do provide guidelines for setting the peak learning rate as a function of other related variables, such as model size, data budget, batch size, and more. These offer valuable priors for determining the optimal learning rate.

We first scan learning rates at five anchor points: three different model sizes and two different D/N ratios. We identify the optimal learning rates at these anchor points and fit a parametric form for the optimal learning rate.

For all model recipes except OLMo-2, we fit a shared exponent a and b, along with a model-specific initial learning rate lr₀. For OLMo-2, we use the optimal learning rate specified by its recipe. We treat OLMo-2 this way because AI2 released a small model scaling law as part of the recipe, which defines the optimal hyperparameters for the scales we study. We also verified that our production learning rate is at or near optimal at the compute optimum of 3.16e18 FLOP.

Key Technical Results

Explain some anomalies in the chart.

We observed that, across both model and data dimensions, computational efficiency has generally improved over time, as expected. Some outliers we observed:

• NeoX performs worse than GPT-2 at 1e19 (although it performs better in the range of 1e17 to 3.16e18). This may stem from noise in the OLMES evaluation. We also observe that NeoX outperforms GPT-2 on the held-out pretraining loss of the FineWeb-Edu corpus.
• The Pile appears to perform significantly worse than OpenWebText. This is not surprising, as the Pile’s main improvement lies in data corpus diversity rather than filtering. It features a carefully curated mixture of 22 sources, including PubMed and arXiv papers, GitHub code, legal opinions, patents, and parliamentary records. For many of these tokens, cross-domain transfer to OLMES (English Web Prose Multiple Choice) may be minimal, resulting in lower computational efficiency. We note that, due to its larger scale, we expect the Pile to eventually outperform OpenWebText (which is indeed much smaller) at larger scales.
• It is also worth noting that the compute multipliers for NeoX and Pile were obtained through extrapolation, introducing additional potential error.

How is the hashrate multiplier calculated, and what are its error margins?

• Each point on the compute scaling curve is derived from multiple independent seed training runs. The error bars represent the standard deviation of OLMES evaluations across these seeds.
• Consider the reference model or data corpus at a given reference performance level under a specific computational power level.
• We then calculate the compute multiplier by finding the leftmost point on the candidate model or corpus’s compute scaling curve where it first reaches the reference performance level. The ratio of the reference compute requirement to the candidate’s compute requirement is the candidate’s compute multiplier.
• The error bars for the hash rate multiplier are derived from a bootstrap of parameters across the entire estimation process and represent a one-standard-deviation interval.
• We indeed wish to emphasize that the actual uncertainty in the model’s compute multiplier is higher than what the error bars suggest. This is because the range of hyperparameter tuning we conducted was limited, and the final performance or held-out loss may be quite sensitive to the exact choices of peak learning rate, batch size, and other factors.

It is also important to note that our ablation studies cannot fully capture the full scope of compute efficiency improvements for several reasons. In fact, from 2019 to 2025, we observed annual year-over-year compute efficiency improvements of 1.24x [1.19, 1.29] on the model side and 1.51x [1.45, 1.57] on the data side. When measured jointly, we observed an annual year-over-year compute efficiency improvement of 1.57x [1.49, 1.65]. This is substantially lower than the average estimate of 3x year-over-year improvement by Anson Ho et al., for the following reasons:

• Many benefits may depend on scale or be particularly important in longer contexts, but our operational scale is too small to capture many of these benefits. For example, layer norms and QK norms in OLMo-2, and parallel attention plus MLP blocks in NeoX.
• In our research, inference efficiency optimizations (such as GQA in Llama-3, a KV cache optimization) do not translate into computational multipliers. We also did not study improvements to tokenizers.
• The hash rate multiplier we obtain is highly sensitive to the model architectures or training corpora we select each year. We have chosen what we believe to be representative model architectures and training corpora, but this does not constitute an exhaustive proof that they are the best for each year.
• We focus on the compute multiplier relative to the OLMES benchmark (which combines ten relatively simple task types), rather than the compute multiplier required to achieve a specific perplexity metric. If we look at other benchmarks—such as those specialized for coding or problem-solving—the numbers would be significantly different, as those benchmarks may reward entirely different data engineering approaches.

We also wish to note that we did not explore improvements on other data sides, such as collecting more high-quality data from new sources, human-expert-generated data, or synthetic data generation methods. Most of the corpora we studied are curated (subsets) from the same Common Crawl, rather than expanding the available data set. This clearly involves consuming a finite stock, and the extent to which this lever can drive progress is limited.

Yield independence from model formulation and dataset

We conducted the following analysis to determine how independent the benefits of model architectures and datasets are. We examined the OLMES score grid at 3.16e18 FLOPs. Performing a linear regression on OLMES scores with the model OLMES score = mean + model effect + dataset effect yielded an R-squared value of 0.88. This indicates that 88% of the variance in OLMES scores can be explained by the additive effects of model and dataset improvements, with only about 12% of the variance attributable to interaction terms or higher-order effects, as well as evaluation noise. This suggests that complex interactions between models and datasets—such as a specific model improvement depending on a particular data engineering approach, or vice versa—are relatively small.

Anson Ho and colleagues estimated that software efficiency improvements (in pretraining) amount to a 3-fold increase per year (95% confidence interval: 1.5-fold to 64-fold). As Ho noted in this blog, “Most software advances may actually stem from improvements in data quality,” and “scaling from a few algorithmic changes related to scale.”

We use the nominal convention C = 6ND to calculate hash rate.

The 2019 model recipe was GPT-2; the 2025 model recipe is OLMo-2. The 2019 dataset was OpenWebText; the 2025 dataset is UltraFineWeb.

We encountered some training instability issues (gradient spikes) with our GPT-3 implementation on The Pile.

These include: optimizer improvements, warmup with decay scheduling, replacing learned absolute positions with RoPE, RMSNorm with SwiGLU-gated MLP, norm reordering, QK-norm, Z-loss regularization, and cleaner initialization.

For the compute scaling chart, we used at least three seeds for each configuration. For the 7x7 grid of model recipes and dataset combinations under a budget of 3.16e18, we used only one seed per configuration.

The 1.57x year-over-year multiplier is calculated using joint improvements from the 2019 model and corpus to the 2025 model and corpus, rather than the product of the 1.24x improvement on the model side and the 1.51x improvement on the data side.