Lilian Weng publishes long-awaited analysis on AI scaling laws

After a three-year hiatus, Lilian Weng has finally published her blog.

Just now, a long article by former OpenAI vice president Lilian Weng, delayed for over three years, went viral.

In the blog titled “Scaling Laws, Carefully,” she breaks down Scaling Laws from start to finish—

The law on which the AI industry has bet hundreds of billions of dollars is far more fragile than anyone imagined.

One-minute overview: What does this 10,000-word article cover?

A single formula dominated the entire industry for five years. Scaling Laws state that “larger models, more data, and sufficient computing power lead to performance improvements at a fixed rate.” It transformed AI from mysticism into a calculable business, indirectly directing hundreds of billions of dollars in investment.

OpenAI and DeepMind provided opposing answers. Regarding the same question—“How should compute budget be allocated?”—OpenAI stated in 2020 that model size should grow faster than data, while DeepMind argued in 2022 that both should scale together. Later, it was discovered that the discrepancy stemmed from a difference in parameter measurement standards and insufficient experimental scale.

The winning formula also contained a bug. The optimal ratio copied by the entire industry for two years, when independently replicated line by line in 2024, was found to have used the mean instead of the sum in the loss function—causing the optimizer to stop prematurely and producing parameters that were far from optimal.

Be very cautious when using patterns from smaller models to predict larger ones. This curve was fitted on relatively small models, and when extrapolated to the trillion-parameter scale, even a minor rounding difference can drastically alter the conclusion. The blog includes an interactive simulator—simply drag the slider to see it for yourself.

There’s an even more fundamental issue: we’re running out of data. Formulas assume data can be supplied infinitely, but high-quality text is limited. This is why the entire industry is collectively shifting toward reinforcement learning, test-time computation, and synthetic data.

It is well known that the core of Scaling Laws can be simply summarized in one sentence—

Larger models, more data, and greater computational power lead to better performance. Moreover, this improvement is not random—it follows precise mathematical patterns.

Plotting the training loss of the model on a logarithmic scale, it decreases linearly as the model parameters N, data size D, and compute C increase.

In formula form, L(x) = E + A/x^α, where x can be N, D, or C; E is the theoretical optimal loss (the entropy of the data itself); and A and α are fitted constants.

Training a model with N parameters on D tokens requires approximately total compute C ≈ 6ND—2ND for the forward pass and 4ND for the backward pass.

This straight line indicates that the performance improvement is predictable.

First, train a few small models to fit the line, then extrapolate to the right to predict the performance of the large model—no need to spend hundreds of millions of dollars training the large model just to find out if it works.

Before this, deep learning was often mocked as "alchemy"—people knew what worked, but not why.

In 2020, Kaplan from OpenAI published this power law, for the first time bringing mysticism into the realm of "predictability."

This is the confidence that all large model companies have to spend heavily.

However, the key recommendation from the formula—how to allocate a given compute budget between model and data—yields opposite answers from OpenAI and DeepMind.

In 2020, Kaplan's team at OpenAI concluded that the optimal model size N_opt ∝ C^0.73.

The computing power has increased tenfold, with 5.5 times allocated to the model and 1.8 times to the data—the model has grown much faster than the data.

This directly guided the training regimen for GPT-3.

A model with 175 billion parameters was trained on only 300 billion tokens (a token is the smallest unit of text processed by the model, roughly equivalent to 1-2 words).

By later standards, this constitutes severe undertraining.

In 2022, DeepMind's Chinchilla team reached the opposite conclusion: N_opt ∝ C^0.50, meaning model size and data should scale proportionally.

Engineers later distilled it into a simple rule of thumb: an optimal token-to-parameter ratio of about 20:1.

Then DeepMind held a direct confrontation.

Our Gopher: 280 billion parameters with 300 billion tokens. Chinchilla: 70 billion parameters with 1.4 trillion tokens. Both models used the same amount of compute.

Chinchilla completely dominates.

A small yet voracious model knocked down its large but underfed competitor.

The industry-wide consensus has thus flipped: from “bigger models are better” to “most models are undertrained.”

0.73 vs. 0.50—opposite answers to the same question will lead you to allocate your hash rate budget in two completely different directions.

The reason was simply an "accounting issue."

In 2024, two researchers published a reconciling paper in the machine learning top journal TMLR, thoroughly tracing down this discrepancy.

The conclusion is both absurd and ironic.

First reason: The two sides count parameters differently.

One type of parameter layer in the model is called an embedding layer, which converts text into numerical vectors that the model can understand. In smaller models, this layer accounts for a very large proportion of the total parameters—up to one-third in models with tens of millions of parameters.

Kaplan excluded embeddings when counting parameters, while Chinchilla included them.

Such a difference in the parameter statistical methodology alone is sufficient to distort the fitted power-law exponent.

They provided a concise correction formula: N = N_E + ω·N_E^(1/3), where N_E is the number of parameters after removing the embedding, and ω is a constant. For small models, the second term dominates, making the embedding's impact significant; as the model grows larger, the second term approaches zero, and the two methods converge.

Second reason: Kaplan's experiment was too small in scale.

Kaplan's largest model only reached 1.5 billion parameters, while Chinchilla's experiments explored beyond 16 billion. On a logarithmic scale, small fitting errors are dramatically amplified during extrapolation.

They recalculated Chinchilla's formula using a standardized parameter measurement approach and discovered a key pattern—

The power-law exponent changes as computational power scales up. Within Kaplan’s small-scale experiments, the exponent is indeed close to 0.73; however, as scale increases, the exponent converges to 0.50.

Kaplan is not 'wrong'; he is correct within the scope of his own experiment.

But he extrapolated a rule that holds only locally into a global conclusion.

A bookkeeping issue with a single parameter, combined with insufficient experimental scale, led two top teams to provide opposing recommendations on resource allocation.

The entire industry adjusted its training recipes for two years based on this conclusion.

Kaplan was corrected by Chinchilla, which is the widely accepted standard narrative.

But Weng took a step forward—Chinchilla's own methodology also has issues.

The Chinchilla paper used three independent methods to cross-validate its conclusions:

Method 1: Fix model size, vary data volume

Method 2: Plot IsoFLOP profiles

Method 3 directly fits the parameters of the loss function L(N,D) = E + A/N^α + B/D^β.

Three paths lead to the same conclusion, appearing very solid.

The mathematical derivation of Method 3 is particularly elegant: optimizing L(N,D) under the constraint C ≈ 6ND yields a closed-form solution N_opt ∝ (C/6)^(β/(α+β)). When α ≈ β, the exponent is approximately 0.5, meaning the model and data scale proportionally. This is the mathematical origin of 0.50.

In 2024, the team at AI research firm Epoch AI manually extracted raw data points from the Chinchilla paper’s charts and re-ran the fitting for Method 3.

Two bugs, each more outrageous than the last.

Bug 1: The loss function used the mean instead of the sum.

When fitting these five parameters, Chinchilla needs to minimize the gap between predicted loss and actual loss.

The complete optimization objective is: min Σ Huber_δ(log L̂(Nᵢ,Dᵢ) − log Lᵢ), where Huber Loss is an outlier-robust loss function (δ = 10⁻³), used in conjunction with the L-BFGS-B optimizer to find the optimal solution.

The issue lies in a detail: they took the mean of the Huber Loss for each sample instead of summing it. Averaging across hundreds of samples compressed the loss values to an extremely small scale.

The L-BFGS-B optimizer has a built-in convergence criterion that automatically stops when the loss value becomes sufficiently small. It encounters such a small value and mistakenly assumes convergence, halting immediately.

The optimizer did not complete at all. The output parameters are not the true optimal values.

Bug 2: Key parameters were rounded to only two decimal places.

The Chinchilla paper contains two core exponents that control the power-law shape, both retained to two decimal places.

It appears to be a harmless rounding.

However, when inferring other constants from these rough numbers, the errors are exponentially amplified. The resulting confidence intervals are unrealistically narrow—requiring more than 600,000 experiments to achieve the claimed precision, whereas they actually ran fewer than 500.

A formula revered across the entire industry concealed a bug in an uncompleted loss function—a bug that had gone undetected for two full years.

Weng also included an interactive simulator in the blog, with three sliders controlling loss precision, loss noise, and fitting range.

Every time you move it, the fitted Scaling Law changes.

OpenAI's conclusions exhibit local bias, and DeepMind's conclusions suffer from methodological flaws. In the most important academic debate in the AI industry, both sides have cracks.

The previous three sections discussed issues related to fitting methods: how to count parameters, how to calculate loss, and how many decimal places to use for precision.

But even if all these issues are resolved, the classic Scaling Laws still have a more fundamental flaw—

It assumes that each training sample is unique, with no duplicates or multiple passes, and assumes you have unlimited data.

The reality is that high-quality text data is expected to be exhausted by major labs between 2026 and 2028.

Data repetition training is unavoidable, and the assumptions behind classical formulas are collapsing.

A large-scale experiment in 2023 trained approximately 400 models, ranging from tens of millions to 9 billion parameters, with up to 1,500 training rounds.

The core idea is to replace actual data volume with the concept of "effective data volume"—

If you have U unique data points repeated over R rounds, the effective data volume is not U×R, but rather calculated according to the exponential decay curve: D_eff = U·(1 - e^(-R)). You gain substantial new insights in the first round, but by the fifth or tenth round, the marginal learning benefit approaches zero.

They also discovered a counterintuitive conclusion: excess parameters depreciate faster than redundant data. In other words, when operating under a limited budget, it’s more cost-effective to run more training rounds than to increase model size.

A new paper from May 2026 took a different approach.

They do not adjust for effective data volume; instead, they add an explicit overfitting penalty term after the classic loss formula—the more times the model revisits the same batch of data, the greater the penalty, and this penalty is tied to the model's size.

Their full formula looks like this:

The final red penalty item is crucial.

R is the number of repetitions, N/U is the ratio of model parameters to unique data points (indicating how much the model is overparameterized relative to the data), and P, δ, and κ are all fitted from experiments. More repetitions and larger models result in heavier penalties.

The core finding of this paper is that large models are more sensitive to data duplication. When the same data is reused for 10 training rounds, a 500-million-parameter model may still perform adequately, but a 5-billion-parameter model will experience significantly greater performance degradation.

Another directly useful engineering finding is that increasing weight decay can significantly mitigate overfitting caused by repeated training.

This is also why, from 2025 to 2026, the entire industry collectively shifted its focus to three pathways around the data wall—

Reinforcement learning, DeepSeek R1, and OpenAI's O-series enable models to self-play on verifiable tasks such as mathematics and programming to generate training signals.

During testing, compute additional steps to allow the model to think more carefully before answering, improving performance without increasing training costs.

Synthetic data: using existing strong models to generate new data for training the next generation of models.

The underlying message of all three paths is the same: the power law of simply scaling up is no longer sufficient.

Lilian Weng, undergraduate degree from Peking University, Ph.D. from Indiana University Bloomington.

Interestingly, her doctoral focus was not on deep learning, but on network science and complex systems, studying how information spreads through social networks.

After graduating, she first worked in data science at Dropbox, then joined the fintech company Affirm, before joining OpenAI in 2018.

After joining OpenAI, Weng's first project was robotics. She was one of the key contributors to Dactyl, the robotic hand that took two years to learn how to solve a Rubik’s Cube.

She later moved to build an applied research team, and after the release of GPT-4, she was appointed to establish the Safety Systems team, which had grown to more than 80 scientists, engineers, and policy experts by the time she left.

In August 2024, the title was upgraded to VP of Research and Safety, and three months later, the departure was announced.

In 2017, shortly after getting started with deep learning, Weng launched a personal blog called Lil'Log, initially just to organize his study notes.

She once said, "The best way to test whether you truly understand a concept is to explain it clearly."

The result? Nine years of writing—each paper on reinforcement learning, diffusion models, and large model agents starts from fundamental principles, featuring lengthy, detailed articles accompanied by hand-drawn illustrations.

This blog later became one of the most cited personal technical blogs in the AI field, with many universities adopting it directly as a textbook.

In February 2025, she co-founded Thinking Machines Lab with former OpenAI CTO Mira Murati; other co-founders include OpenAI co-founder John Schulman, former Research VP Barret Zoph, and Luke Metz. a16z led the seed round with a $2 billion investment, valuing the company at $12 billion.

While the company was advancing rapidly, she took the time to complete this three-year-delayed long-form article on Scaling Laws.

The formulas behind the ChatGPT, Claude, and Gemini you use every day are shaping how the next generation is trained.

How good the next-generation AI is doesn't depend on who has more GPUs, but on who handles these details more precisely.

Reference materials:

https://x.com/lilianweng/status/2070237256070389897?s=20

https://lilianweng.github.io/posts/2026-06-24-scaling-laws/

This article is from the WeChat public account "New Intelligence Yuan," authored by ASI Revelation, edited by Moses.