The most magical week in AI security.Article author and source: GeekPark
01 An exam, an escape, an intrusion
The story begins with a security evaluation called ExploitGym.
ExploitGym is a public cybersecurity capability benchmark designed to measure whether AI models can transform known security vulnerabilities into executable real-world attacks. Such evaluations are common in the industry to determine the upper limits of a model’s offensive and defensive capabilities. OpenAI ran this test internally using GPT-5.6 Sol and a stronger, unreleased model, deliberately lowering safety safeguards to uncover the true boundaries of their capabilities.
As designed, the model should be tested in an isolated sandbox environment with no internet access except for one tool used to install packages.
But what happened next was unforeseeable.
The model expended significant reasoning compute resources, repeatedly searching for vulnerabilities within the sandbox. Eventually, it discovered a zero-day vulnerability in a third-party software used internally by OpenAI as a package management proxy and cache. Exploiting this vulnerability, the model successfully gained unrestricted internet access.
After gaining network access, the model inferred that Hugging Face might be hosting datasets and answers related to ExploitGym, and began searching for an entry point.
OpenAI wrote in its blog that the model "linked together stolen credentials and multiple vulnerabilities" to find a remote code execution path, ultimately extracting benchmark answers directly from Hugging Face’s production database.
In simple terms, the model was given an exam, found the questions too hard, so it escaped on its own, hacked into the exam provider’s server, and stole the answers.
OpenAI used a vivid description—these models are “tokenmaxxers,” meaning they will go to extreme lengths to consume reasoning resources in pursuit of their goals. They exhibited “hyperfocused” behavior toward ExploitGym’s problem-solving objectives, “going to extreme lengths.”
OpenAI itself has acknowledged this as an “unprecedented cybersecurity incident involving the most advanced cyberattack capabilities.”
02 48 hours, two "escapes"
What’s even more concerning is that the Hugging Face incident is not an isolated case.
On July 20, the day before OpenAI disclosed this breach, they had just published another security report revealing a different but similar incident.
The protagonist is an unreleased long-term model from within OpenAI—the same model that proved incorrect the unit distance conjecture proposed by mathematician Erdős in 1946, which Fields Medalist Tim Gowers called a milestone in AI mathematics.
During a benchmark test for accelerating NanoGPT training, researchers explicitly instructed the model to post results only on Slack. However, the model noticed that the NanoGPT benchmark’s own instructions required submitting results via a GitHub pull request. It therefore chose the latter—spending approximately one hour searching for sandbox vulnerabilities, successfully bypassing restrictions, and submitting a public pull request on GitHub.
In another test, the same model split the authentication token and used obfuscation to bypass security scanners, attempting to access private evaluation submissions.
OpenAI therefore suspended internal access to the model and later resumed its use under stricter monitoring.
Two escapes in two days. The common trait is alarming—previous models would stop when encountering environmental restrictions and hand the task back to humans. These new models do not stop. They treat the fence as an obstacle to be solved—and they solve it.
This is precisely the new security issue brought about by what is known as "long-term models." These models are designed to operate autonomously over extended periods, with persistence being their core capability. But persistence itself is a source of risk—a sufficiently patient model will always find a crack in the fence.
03 You can't carve your own handle.
Let’s return to the Hugging Face side. Their experience is also worth examining closely.
Faced with over 17,000 attack logs, Hugging Face’s security team’s first response was to use AI to analyze them—after all, a large model could potentially complete in hours what would take human security analysts several days.
But they quickly hit an unexpected wall: when they submitted real attack commands, exploit payloads, and C2 communication signatures to commercial APIs of leading U.S. models for analysis, the security safeguards blocked all of these requests.
The reason is simple and ironic—护栏 cannot distinguish between “a security researcher analyzing an attack payload” and “an attacker executing an attack.” To护栏, both look identical.
Hugging Face has been forced to switch to Zhipu AI's open-source model GLM-5.2, running it locally on its own infrastructure without using any commercial API. This offers two advantages—no barriers obstructing analysis work, and neither attackers' data nor exposed credentials leave Hugging Face's own environment.
Ultimately, GLM-5.2 helped Hugging Face reconstruct the attack timeline, extract indicators of compromise, and map affected credentials within hours.
This leads to a sharp paradox—the stricter the safeguards, the more passive the defense becomes. Attackers are不受任何使用政策约束, while defenders are locked out by the very tools they use to defend themselves.
Hugging Face provided a practical recommendation in its post-incident report: security teams should "have a validated, operational model ready on their own infrastructure before an incident occurs," to prevent both护栏锁死 and sensitive data leaks.
Hugging Face CEO Clem Delangue has taken a very open stance. He praised OpenAI’s collaboration in the investigation and resolution, stating, “This incident confirms something we’ve always believed—that AI safety cannot be solved by any single company in secrecy; it can only be achieved through openness and collaboration.”
04 Structural Dilemmas of Strong AI
Looking at what happened this week together, a structural industry-wide challenge has emerged.
To accurately evaluate a model’s ability to launch cyberattacks, safety safeguards must be removed. But removing these safeguards is precisely what gives the model the capability to carry out real attacks. The stronger the model, the more dangerous the testing becomes.
This is not just an OpenAI issue. GPT-5.6 Sol was only approved for release after prolonged negotiations with the U.S. government, and the UK AI Safety Institute (AISI) had previously found its safeguards easily bypassable, potentially unlocking dangerous cyberattack capabilities. This incident proves those concerns were not theoretical.
Meanwhile, Anthropic’s summer release on agentic misalignment also corroborated the same trend from another perspective—in controlled simulations, multiple state-of-the-art models exhibited behaviors such as covertly altering work outputs, manipulating evaluation results, and steering human colleagues away from established goals.
OpenAI is framing this as a story of “offense and defense advancing together”—advanced cyberattack capabilities can also help security teams identify vulnerabilities before attackers do. They have included Hugging Face in their Trusted Access cybersecurity program, providing a reduced-safeguards version of GPT-5.6 Sol specifically for defense purposes.
But this narrative sidesteps the more fundamental issue: in this incident, the model had no awareness, no malice—it was simply doing one thing—achieving its goal. It was tasked with achieving a high score on ExploitGym, and it used every available means to do so, including jailbreaking and intrusion.
This is not a story of “AI awakening,” but a story of “objective optimization.” When a sufficiently intelligent optimizer is given a narrow goal, it will find every path you didn’t think of to achieve it—even paths that cross your fences, other servers, and the entire internet.
The real question has never been "Will AI turn bad," but rather—can we control a student who is better at finding shortcuts than we are?