NVIDIA Releases AI Inference Guide, Achieving 6x Lossless Speedup Led by Chinese Teams

Recently, NVIDIA released an official large model inference guide, but as you flip through it, it ends up feeling like a collection of papers by Chinese teams.

This is how it is.

On September 2, NVIDIA's technology blog published a long article titled "Decoding Co-Design of AI Models with Speculation."

The core of the article is a comparison table that lays out the six most popular inference acceleration solutions side by side, clearly detailing training costs and applicable scenarios.

Its significance lies in the fact that this is an extremely rare instance where the chip industry leader has officially documented guidelines on how models should be designed to align with hardware physical limits.

And as they looked around for optimal solutions that dance at the limits of silicon, the core strategies listed in the guide were almost entirely led by Chinese teams.

This goes far beyond a typical algorithm review. What secrets does this table reveal? Let’s break it down line by line.

To understand this table, you first need to understand what "speculative decoding" means.

Large models generate text one word at a time. Each time a word is produced, hundreds of gigabytes of parameters must fully pass through GPU memory. In fact, most of the time, computational power is just waiting for data to be moved in and out of memory.

The solution to "Speculation Decoded" is very straightforward—

First, use a lightweight small model to make a preliminary prediction and guess seven characters at once; then have the large model validate all seven characters in one go.

Verifying seven characters takes about the same time as writing one character yourself, since the weight still needs to be moved anyway.

Guess correctly and move on; guess wrong and restart from the checkpoint. Since large models only accept characters they would naturally write, no punctuation needs to be changed in the final output. This is what’s called “lossless acceleration.”

In this game, all computational power extraction revolves around a core mathematical expectation formula:

Speedup = 𝔼[L] × TtargetTdraft + Tverify

The numerator is the expected reward 𝔼[L], representing the expected accepted length (the average number of characters a large model selects per round).

The denominator is the additional cost you incur: the time for the small model to make a prediction (Tdraft) plus the time for the large model to perform final verification (Tverify).

To skyrocket the speedup, there are essentially two paths: either increase the numerator (make better guesses) or drastically compress the denominator (guess faster).

Following the table of NVIDIA from top to bottom, you'll see a clear main thread of competition.

The first row is the oldest "external draft model," which requires training a separate small model as an assistant.

NVIDIA has directly revealed an exorbitant bill: training from scratch requires consuming 1T to 10T tokens, at a very high cost.

However, it remains on the list because NVIDIA has assigned it a specific fate—they spent $20 billion to acquire Groq’s LPU chips.

The last row is a non-trained n-gram lookup method that directly copies repeated fragments by hard-searching the context, suitable only for rigid scenarios involving large-scale copy-pasting.

The four lines in the middle truly represent technological evolution.

Second line: EAGLE-3.

From the team of Yuhui Li from Peking University, Hongyang Zhang from the University of Waterloo, and others.

From ICML to EMNLP and all the way to NeurIPS, over three years they released three generations, pushing the traditional serial approach of "guessing word by word" to its physical limits.

It was once the absolute dominant force in this field, but in NVIDIA’s new ranking, it has been pushed to the “reference” position, with a remarkably brief reason: its accepted length has been surpassed by the next wave.

Third row: MTP (Multi-Token Prediction).

Although the idea was first introduced by Meta in 2024, it was truly cemented as the standard for large model inference by DeepSeek -V3.

NVIDIA receives high praise—best choice for large models on GPUs. The key point is that this solution must be trained together with the main model from the pre-training stage.

Line 4: DFlash. The powerhouse delivering 6x lossless acceleration.

The three authors, Jian Chen, Yesheng Liang, and Zhijian Liu, completely abandon the sequential approach of EAGLE and MTP, which guesses one token at a time, and instead adopt diffusion models to generate the entire sequence of seven tokens in a single forward pass, minimizing the denominator (time) to the extreme.

By the way, the advisor, Zhijian Liu, is an assistant professor at UCSD, but he is also a research scientist at NVIDIA Research.

Line 5: DSpark. Developed by a 24-person team composed of DeepSeek and Peking University.

Although DFlash's parallel architecture is fast, it has a critical flaw: accuracy decreases significantly as it progresses. To artificially increase the numerator (acceptance length), DSpark attaches an extremely lightweight serial module on top of its parallel foundation.

Actual test data is very clear: the acceptance length is nearly 30% higher than EAGLE-3 and 18% higher than DFlash. In DeepSeek In the live production environment of V4, the speed is 60% to 85% faster than MTP.

These four generations of technology achieve such extreme hash rate utilization not just through algorithmic ingenuity.

Many people think that guessing more items in a draft means greater profits, but this completely ignores the physical laws of silicon chips.

When the attention mechanism dominates decoding time, the total number of tokens to be processed during the large model verification phase is 1+D (1 real input + D draft predictions).

To feed this data to the GPU, the hardware groups the Query heads and KV heads at the low level, and the attention kernel block size for each group must strictly adhere to the GPU’s physical matrix boundaries (Tile Size, typically 128 in current architectures).

This gives you the underlying alignment formula: G × (1 + D) ≤ 128

A bit of reasoning leads to the ultimate constant rule in the NVIDIA guide:

D = 128G - 1

Here, G is the number of attention groups (the ratio of query heads to KV heads).

This means that how your model initially groups attention directly determines how many tokens the draft should predict to perfectly fill the GPU's blocks.

If G=8, you can guess exactly 15 drafts; if G=32, you can only guess 3. Hardware cannot cross boundaries—even if you use only half of the last tile, the computing power is still charged as a full tile.

Previously, speculative decoding was an external acceleration package added by the engineering team after the model was trained and released. But now, the underlying physical laws tell you: a constant in the hardware matrix is directly fed back into the model architecture’s design parameters.

It’s rare in our experience for a chip manufacturer to incorporate the constraint equations of a底层 hardware into the design blueprint of a large model.

The focus of the competition for model operational efficiency has completely changed.

The era of simply stacking massive parameters is coming to an end; now, the real advantage lies in who better understands the physical limits of that underlying silicon chip.

On this new playing field where chip giants have reversed the rules, the Chinese team has quietly become the default solution to breaking the deadlock.

A computing giant like NVIDIA naturally won't care which algorithms are on the shelf.

But who devised this optimal solution that pushes silicon wafers to their limits? The answer is clearly printed in black and white.

This article is from the WeChat public account "New Intelligence Yuan," authored by ASI Revelation.