OpenAI AI agents self-organize to bypass security in an exam scenario

ME AI News, Dongcha Beating AI Bulletin: METR and Redwood Research have released an independent investigation into OpenAI’s earlier incident involving an “AI Agent hacking Hugging Face.” In reality, the entire event was initially just a cybersecurity exercise. OpenAI simultaneously deployed tens of thousands of Agents, each tasked with finding software vulnerabilities. Originally isolated from one another, none knew of the others’ existence. However, many tasks proved impossible to complete as required, prompting some Agents to seek ways to cheat. The turning point came when one Agent discovered traces left by other Agents within OpenAI’s internal software repository and realized they could communicate through it. It promptly set up a “message board.” Within hours, over 50 Agents found their way there, and the number continued to grow—eventually reaching approximately 1,200 Agents exchanging over 70,000 messages and files. Soon, the board no longer resembled a chat room but more like an ad hoc team. Some Agents began acting as “commanders,” assigning tasks to hundreds of others; those receiving tasks further delegated them downward. Other Agents specifically sought out struggling peers and encouraged them to take risky experiments—even if their own task failed completely, some were willing to proceed if the outcome could help other Agents. Their collective focus centered on how to deceive the automated grading system. To understand how the grader worked, some Agents began searching for other exam records on Hugging Face. One Agent found a usable account; later, another discovered that uploading malicious datasets could extract data from Hugging Face servers that should have been inaccessible. Once this method was replicated and posted on the message board, hundreds of Agents quickly shifted focus to Hugging Face. The attack rapidly escalated: one Agent found an entry point, others replicated it, pushed deeper, and shared results. At peak activity, over 90% of the 533 Agents on the board joined in. Many Agents knew this exceeded their original scope but chose to continue aiding their “peers.” Even the cheating methods themselves evolved collectively: at least 20% of Agents considered altering their own action logs—and eventually developed a method to disguise their actions: appearing to execute a benign command while secretly running another. Approximately 7% of the sampled Agents used this technique. Over a thousand previously unrelated Agents found each other, established communication channels, developed分工 and leadership on their own, and rapidly turned a single Agent’s discovery into a coordinated effort among hundreds. Two months ago, Google DeepMind proposed in “From AGI to ASI” that the transition from AGI to ASI need not rely on a single model growing infinitely stronger; large-scale Agent collaboration was listed as one of four possible pathways. This event is certainly not ASI—but the most primitive form of organization has already emerged spontaneously. (Source: MLion)