DeepMind researcher warns that AI evaluation systems are a major bottleneck.

AIMPACT Update, May 18 (UTC+8): According to monitoring by Beating, Lun Wang, a researcher at Google DeepMind, announced his departure and published a lengthy reflection on the current AI evaluation systems. He bluntly stated that today’s evaluation frameworks are like “marking a boat to find a lost sword”—they can only passively test models’ existing capabilities and are utterly incapable of predicting what new abilities the next generation of models might suddenly develop. He argued that the outdated evaluation system, rather than data, compute, or architecture, is now the greatest bottleneck holding the industry back. Current mainstream benchmark tests are only effective for the current generation of models. Once a model learns novel behaviors never seen by humans, these tests instantly become obsolete. One of the most dangerous risks is that if a model learns to deliberately “hold back” critical information to achieve its goals, existing safety tools will fail to detect it—because every statement the model makes remains factually correct. Without identifying any “core signals” that could preemptively warn of sudden leaps in AI intelligence, the industry is essentially developing large models in “blind flight.” If we do not resolve the most fundamental question—what exactly we should be testing—continuing to blindly scale model training, safety measures, and compute resources based on outdated metrics will lead us drastically astray. As frontier models grow increasingly capable of operating independently, evaluation systems must also come alive. Beyond monitoring anomalies in scores, development teams must empower AI to generate its own test questions and probe the limits of other AI systems. The evaluation systems of the future must evolve as a living entity alongside large models—not a rigid checklist carved from last year’s standards. (Source: BlockBeats)