Anthropic Accuses Chinese AI Giants of Model Distillation, Sparking Global Debate

Author: duduhappy

Original Title: A Comprehensive Guide to the Controversial Technique of Large ModelDistillation

On September 8, the U.S. NSA, FBI, and CISA jointly issued a security advisory targeting Chinese AI companies.

The announcement claims that Chinese AI companies are conducting "industrial-scale distillation" of advanced U.S. models, systematically extracting the capabilities of U.S. models to train their own, and directly framing this as a national security issue.

Subsequently, Anthropic disclosed new findings accusing Chinese companies including Alibaba, Moonshot, and DeepSeek of extensively invoking Claude in various ways to transfer its capabilities to their own models.

China denies the allegations, arguing that distillation is a long-established, general technique in the AI industry, and accuses the U.S. of applying “double standards” to suppress China’s AI competitiveness. Thus, a common technical term that had been included in machine learning textbooks a decade ago has suddenly become a new front in the U.S.-China AI rivalry:

Model distillation.

The issue has also become very acute:

Training a smaller model using a larger model is called "knowledge distillation,"

Why could this behavior by competitors potentially become "model theft"?

To answer this question, don’t rush to take a side—look at it from first principles: what distillation truly solves is just one problem:

If one model has already learned how to solve the problem, why should a second model learn it from scratch again?

This is the starting point of distillation.

Many people think the most expensive part of large models is the GPU, but the GPU is just a surface-level observation—the real cost lies in:

Let the model discover patterns on its own from vast, chaotic data.

The internet contains news, papers, code, novels, advertisements, misinformation, and duplicated content; the model must consume enormous computational power to gradually learn how to analyze different types of questions, how to derive solutions to math problems, where to look for code bugs, and how to structure an article—essentially, this is an extremely costly process of knowledge discovery.

But here’s the question: if a super model has already completed this exploration, do newcomers need to go through it all again? Not necessarily.

This is the first principle behind distillation.

For the simplest example, a photo of a husky might be manually labeled to tell the model only:

This is a dog.

But a strong model might tell it: dog 70%, wolf 25%, cat 4%, fox 1%—the information content is entirely different; the former simply tells you the answer, while the latter also tells you:

Dogs are similar to wolves but quite different from cats.

This is the core idea of classic knowledge distillation: the Teacher does not simply hand over the answer to the Student, but conveys its own judgment across the entire problem space. Therefore, what distillation truly accomplishes is not copying the Teacher’s parameters, but:

Make a smaller, cheaper model approximate the behavior of a stronger model as closely as possible.

Traditional distillation typically occurs within a single company, where the teacher is owned internally, allowing you to see its probability distribution, intermediate layers, or even full parameters—but what’s truly sparked controversy today is:

The teacher isn't yours at all.

You only have one API, what do you do? The answer is straightforward:

Keep asking it.

Prepare hundreds of thousands or even millions of questions: how to solve math problems, how to write code, how to reason through logic puzzles, how to revise articles, have strong models answer them, then collect these “questions + answers” to train your own model—so today, many so-called “large model distillations” have essentially become:

Strong model generates data → Small model learns from the data → Imitates the strong model’s capabilities.

Strictly speaking, this is no longer exactly equal to classical knowledge distillation; it is closer to:

Synthetic data + imitation learning + fine-tuning.

But the industry still commonly refers to it as Distillation, and it is precisely here that the real controversy begins.

This is the most counterintuitive part of distillation: a super model may have seen trillions of tokens and consumed enormous computational power to become intelligent—why can a small model significantly improve by seeing just hundreds of thousands or millions of high-quality data samples? The answer is:

Different information density.

Raw internet data is like "ore," while data generated by powerful models is like already-refined "concentrate"—for example, for a math problem, ordinary data might only contain:

Question + Answer.

Strong reasoning models can provide: reasoning, steps, failed attempts, corrections, and the final answer—effectively re-encoding the expensive, hard-won insights of the Teacher into data that the Student can more easily absorb. Thus, the most important sentence to understand about distillation is:

Distilling true compression isn't just about model parameters—it's about rediscovering the exploration cost required to attain these capabilities.

Previously, the main thing distilled was:

Answer.

Currently蒸ing:

Solution process.

DeepSeek-R1 is a prime example: DeepSeek officially released multiple R1 Distill models, some of which were further trained using approximately 800,000 samples curated by R1, leading to a significant shift in AI training.

In the past:

Human → Data → AI

It's becoming more and more like:

Super AI → High-quality data → Small AI

The strongest model not only serves users, but is also beginning to become:

The teacher training other models.

This issue cannot be judged by technical names alone; you need to consider at least four things.

First, who is Teacher?

Self-training with oneself involves almost no controversy; open-source models explicitly permit derivative training and are relatively straightforward. The truly sensitive issue lies with proprietary competitive models.

Second, what did you receive?

Normal usage of dozens of outputs is clearly not on the same scale as systematically extracting millions or even billions of responses through a large number of accounts, proxies, and automated tools.

Third, is the other party permitted?

Paying for API access does not grant authorization to “infinitely replicate model capabilities”; for example, OpenAI’s current service agreement explicitly prohibits using its outputs to develop AI models that compete with OpenAI’s products and services.

Fourth, what did you ultimately train?

A corporate customer service classifier and a general large model designed to directly replace a teacher have entirely different business implications, so:

Is distillation stealing? This is itself an oversimplified question.

What should really be asked is:

Who, by what means, under what authorization, obtained what information, and used that information to train which model?

The logic behind U.S. model companies is easy to understand: they invested billions of dollars, GPUs, and top talent to develop these model capabilities; if competitors can easily replicate them via API at low cost, it essentially bypasses the most expensive R&D process. But on the other hand, there is also a sharp issue:

These foundational models have also been trained on vast amounts of human-written articles, code, images, and internet content, so:

The process of a model learning from human creations is called "training".

When did another model learning from model outputs become “theft”?

This is why the controversy surrounding distillation will ultimately extend beyond technical matters, touching on: copyright, contracts, API usage rules, unfair competition, model licenses, and even technological competition between nations.

Some might think: If that’s the case, can’t I just keep calling GPT to replicate it? It’s not that simple—the black-box API exposes only:

Finite answers to finite questions.

You cannot see the full training data, parameters, and internal representations, nor can you enumerate all possible questions, so what the Student truly learns is usually not the complete Teacher; more accurately:

What it learns is a projection of the Teacher within your sampling range.

Asking a million coding questions can train a strong coding model, but that doesn’t mean it automatically gains the teacher’s full mathematical, scientific, writing, and generalization abilities—this is precisely why the most advanced foundational models still hold immense value.

The most important issue in the AI industry in the past was:

How much does it cost to train the strongest model?

The increasingly important question going forward may become:

After the strongest model has been developed, how much would it cost to replicate 70% or 80% of its capabilities?

If creating initial capabilities requires tens of billions of dollars, but migrating those capabilities requires only a small fraction of that, the entire AI industry will see new specialization:

The largest model handles exploring capability boundaries, the medium-sized model handles承接 and distillation, and the smallest model enables low-cost, large-scale deployment.

This is why distillation is gradually becoming a core variable in the global AI competition, evolving from a model compression technique.

Understanding distillation, it's easiest to fall into two extremes: one believes that

Distillation is plagiarism.

Another perspective:

Since the output is accessible, it doesn't matter how you steam it.

Both are too simplistic; distillation itself is merely a capability transfer technique—the true determinant of the boundary is:

Who owns the teacher, how is the data obtained, whether the rules permit it, and what the student ultimately does with it.

But one thing is becoming increasingly clear: in the future AI world, the most expensive thing may not be running an intelligence, but:

Creating a new intelligent capability for the first time.

Because once this capability is created, synthetic data, distillation, and small models will rapidly spread it throughout the entire industry, what distillation truly "squeezes out" is not parameters, but:

The exploration cost has already been paid by predecessors.

Twitter: https://twitter.com/BitpushNewsCN

BitPush Telegram community: https://t.me/BitPushCommunity

BitPush TG subscription: https://t.me/bitpush