Just now, DeepSeek V4 was updated.
We have newly launched the speculative decoding framework DSpark and simultaneously open-sourced DeepSpec, the full-stack speculative decoding framework supporting this version.
DeepSeek-V4-Pro-DSpark is not a newly architected model, but rather builds upon DeepSeek-V4-Pro by integrating a speculative decoding module. This update focuses on engineering implementation rather than advancements in the model's core capabilities.
DSpark has been deployed in live traffic for DeepSeek-V4 (Flash and Pro), significantly accelerating inference for large language models (LLMs).
Technical Report: "DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation"
Technical report link: https://github.com/deepseek-ai/DeepSpec/blob/main/DSpark_paper.pdf
The core objective of DSpark is to address latency and throughput bottlenecks in LLM inference within production environments, particularly under high-concurrency scenarios. In short, DSpark successfully combines high-throughput "parallel generation" with adaptive "load-aware verification."
Speculative decoding is a technique that accelerates the inference of large language models without altering the model's output distribution. Its core idea is to introduce a lightweight "draft model" that pre-generates several candidate tokens, which are then jointly verified and accepted by the target model. This transforms sequential, token-by-token generation into parallel batch verification, significantly reducing end-to-end latency.
On this basis, DSpark's innovation lies in introducing a semi-autoregressive generation architecture: it retains the high throughput advantage of the parallel draft model while incorporating a lightweight serial module to model dependencies between tokens within a block, thereby mitigating the acceptance rate decay commonly observed at later positions in parallel draft models.
In addition, there is hardware-aware confidence-scheduled verification: traditional speculative decoding typically sends all generated draft tokens for verification blindly, and during high system load, these tail tokens—most of which are likely to be rejected—waste valuable batch processing capacity. DSpark introduces a confidence head to evaluate the likelihood of each token being accepted. Combined with a hardware-aware prefix scheduler, the system dynamically tailors the optimal verification length for each request based on real-time engine throughput characteristics, allocating computational resources exclusively to tokens with the highest expected return.
To deploy in real online infrastructure, DSpark’s scheduler employs an asynchronous mechanism to support Zero-Overhead Scheduling (ZOS) and continuous CUDA graph replay. It leverages historical predictions from the first two steps to determine the current dynamic truncation length, thereby hiding scheduling latency, preventing GPU pipeline stalls, and ensuring complete lossless reconstruction of the target model’s output distribution.
In tests covering areas such as mathematical reasoning, code generation, and everyday dialogue, DSpark significantly outperforms current state-of-the-art autoregressive models (Eagle3) and parallel draft models (DFlash). For instance, on the Qwen3 series (4B, 8B, 14B) target models, its average acceptance length improves by 26.7% to 30.9% over Eagle3 and by 16.3% to 18.4% over DFlash.
Compared to the previous generation’s single-token production benchmark (MTP-1), DSpark increases user generation speeds by 60%-85% (Flash model) and 57%-78% (Pro model), while maintaining the same overall throughput.
Along with DSpark, DeepSpec has also been open-sourced—a full-stack codebase for training and evaluating draft models for speculative decoding. It serves as the open-source infrastructure for this approach and other cutting-edge algorithm implementations, including data preparation tools, draft model implementations, training code, and evaluation scripts.
DeepSpec divides the entire process into three stages: data preparation, training, and evaluation. The stages must be executed in sequence, with the output of each stage serving as the input for the next.
During the data preparation phase, you must download the prompt data, use an inference engine to regenerate answers for the target model, and build the target cache. Notably, with the default Qwen/Qwen3-4B configuration, the target cache can reach approximately 38 TB in size; adequate storage resources must be evaluated before use.
The training phase can be initiated by running bash scripts/train/train.sh. This script invokes train.py and launches one worker for each visible GPU. Users can select different algorithms and target model configurations from the config/ directory by specifying config_path. The project also supports adjusting training settings by overriding config_path and target_cache_dir, as well as modifying individual configuration fields using --opts.
On the hardware side, DeepSpec's default configuration and scripts are designed for a single-node, 8-GPU environment. If fewer GPUs are available, users must accordingly reduce the number of visible GPUs in CUDA_VISIBLE_DEVICES.
The evaluation phase is initiated using the bash script scripts/eval/eval.sh. The evaluation script uses the trained draft model checkpoint to measure acceptance rates across multiple speculative decoding benchmark tasks. The evaluation datasets currently listed in the project include GSM8K, MATH500, AIME25, HumanEval, MBPP, LiveCodeBench, MT-Bench, Alpaca, and Arena-Hard-v2, covering diverse task types such as mathematical reasoning, code generation, dialogue capability, and comprehensive question answering.
In terms of algorithms, DeepSpec currently supports three draft models: DSpark, DFlash, and Eagle3. For target model families, the project currently supports Qwen3 and Gemma.
The open-sourcing of DeepSpec consolidates speculative decoding—an engineering practice previously scattered across various research teams—into a reproducible, scalable, and standardized toolchain. For researchers and engineers seeking to accelerate inference for their own large models, this means they can directly train custom draft models on established frameworks, bypassing the need to rebuild repetitive infrastructure.
Reference link:
https://github.com/deepseek-ai/DeepSpec/blob/main/DSpark_paper.pdf
https://github.com/deepseek-ai/DeepSpec
This article is from the WeChat public account "Machine Heart" (ID: almosthuman2014), authored by Zeanan and Yang Wen.