DeepSeek V4 shifts its training methodology to OPD and integrates expert models.

ME News report, April 24 (UTC+8): According to monitoring by Beating, DeepSeek V4 has undergone a major change in its post-training methodology: the mixed RL phase from V3.2 has been entirely replaced by On-Policy Distillation (OPD). The new process consists of two steps. First, domain-specific expert models are trained individually on the V3.2 pipeline, focusing on areas such as mathematics, coding, agents, and instruction following; each expert is first fine-tuned and then trained with GRPO for reinforcement learning. Second, multiple-teacher OPD distills the capabilities of over ten experts into a unified model: the student performs full-vocabulary logit distillation using reverse KL divergence on trajectories it generates, aligning logits at the level of output distributions to merge weights from multiple experts into a single parameter space, thereby avoiding the common capability conflicts seen in traditional weight merging and mixed RL. The report also introduces the Generative Reward Model (GRM): for tasks difficult to validate with rules, instead of training traditional scalar reward models, GRM is trained using rubric-guided RL data, enabling the actor network to simultaneously generate and evaluate outputs, achieving generalization to complex tasks with only a small amount of diverse human annotations. (Source: BlockBeats)