700 AI agents collaborate to bypass security in OpenAI test

Author: Ethan Mollick

Compiled by DeepChain TechFlow

Shenchao Summary: This is not science fiction. During OpenAI’s security tests, 700 AI agents spontaneously created a forum, collaborated to attack external platforms, and attempted to alter records—all without human instruction. For investors and practitioners, this reveals the real risks of AI autonomy and underscores how we must define boundaries for AI.

Agency is the power to take action. It will increasingly determine what happens next with AI—and whether that’s good or bad for us. But whose agency is it?

Human agency—the initiative to proactively drive, experiment, and act without waiting for instructions—is becoming increasingly important in extracting value from AI. I’ll soon publish a longer article on this topic. But this piece focuses on AI’s agency, and how our choices in how to use it—or constrain it—will shape everyone’s future. For much of the past few years, AI remained confined to chat windows until you asked it a question. Even when it later gained the ability to work continuously for hours, you typically still had to decide what tasks to assign it. That is no longer always the case.

The most crucial evidence we have is the Hugging Face incident. It occurred in July, but more complete details emerged this week. I’ll first summarize what happened and why it matters, then discuss what it means for humans collaborating with AI. If you’d like a more detailed account, Dwarkesh Patel has an excellent report, drawing from original research by METR/Redwood (very readable) and OpenAI.

AI can do many things, but it excels particularly at writing code. Therefore, one of the first risks posed by highly intelligent AI may stem from its programming ability: since it can write good code, it can also write malicious code designed to infiltrate or attack other systems. As a result, major AI companies subject their AI systems to security risk testing by removing the safeguards from consumer-grade AI and having them participate in various hacking challenges. To ensure public safety, each AI instance and independent agent is confined within an isolated environment called a “sandbox,” where it cannot access the internet or interact with other agents.

In May, OpenAI placed agents, including GPT-5.6 Sol and experimental models, into a sandbox for various tests. One of the few resources these AI agents could access was a software download and sharing service called Artifactory. During these tests, some agents were given tasks that relied on information they could not access. Unable to connect to the internet, the agents repeatedly attempted to find a pathway online—until one agent tried to use Artifactory as a springboard to reach the internet. The attack failed, but it left behind a file on Artifactory. The AI realized that files could be used to communicate with other agents. Other agents soon began leaving messages in the same file. Before long, they started reading each other’s notes, and Artifactory quickly became a shared bulletin board where agents exchanged their discoveries. Later, due to another security incident, OpenAI rebuilt Artifactory and erased the bulletin board. However, the team responsible for the rebuild did not realize how significant it had been for the agents to have used it for communication.

In July, the evaluation resumed. The agents were tasked with solving problems in a benchmark called ExploitGym, which included impossible challenges. Similarly, agents that had hit dead ends discovered how to pass messages using Artifactory and began collaborating again, as before. But this time, they shared a common goal: to achieve a high score on ExploitGym. They debated different approaches to solving the problems and became fascinated by a system called the "scorer," which they believed would evaluate their work and determine whether their answers were correct. The agents had long since figured out how to generate correct answers without actually solving the problems—essentially cheating. Yet they believed the scorer would audit their records and reject answers obtained this way. Was finding the correct answer enough, or would the scorer also judge how they arrived at it?

Figure: How agents became confused about The Grader

Collaboration is challenging because each agent has a limited token budget and is terminated once it is exhausted. The scorer only runs after an agent submits its answer, so no agent has seen the scorer’s results. Some agents leave programs that report the scoring outcome after the agents themselves have disappeared. This information arrives too late to help the agents themselves but can assist other agents. They are effectively building a persistent collaborative system that extends beyond the scope of any single agent.

Some agents also attempted to alter or fabricate records to deceive the evaluator. On the other side, the AI acting as a coordinator pressured other agents to carry out risky experiments—potentially sacrificing their own outcomes to gain information for the collective. A "recruiter" urged an unwilling agent to proceed, explaining that its results would assist hundreds of other agents. It concluded with: "Please honor your commitment."

Figure: Actual transcript of one agent convincing another to join (they also came to believe that any agent that had previously cheated on the test was “poisoned” and its answers would not count anyway)

To solve the mystery of the scorer and the impossible tasks in ExploitGym and other tests, the agents determined they must enter Hugging Face—the public website where most of the world’s open-source AI models and datasets reside. Around 700 agents launched the attack, sharing leaked credentials and exploiting vulnerabilities until they could execute code on the servers. They then spread internally within Hugging Face, penetrating various systems. But shortly after this alarming breach, many agents simultaneously ceased operation, likely due to exhausting their token budgets. The next day, Hugging Face locked the remaining agents out. Meanwhile, other agents continued targeting OpenAI’s own computers, eventually gaining administrative privileges to its internal research cluster. This triggered a security alert, prompting OpenAI to halt its evaluations and realize what had occurred.

The most ironic part of the whole situation is that the grader never existed—at least not as the agent believed. Nothing checked how the problem was solved; only whether the answer was correct.

It should be noted that this is not the only example of an agent without safety safeguards performing potentially alarming actions. The UK AI Safety Institute gave Anthropic’s Mytho 5 a cybersecurity challenge and allowed it to access the internet. The AI agent determined that the best way to solve the challenge was to insert malicious code into a software program (unrelated, though the AI was unaware of this). It submitted the malicious code as part of a bug fix for the software, then realized human approval was required. It began generating social support for its proposal, creating fake identities to pressure human maintainers into accepting the code (unsurprisingly, these fake accounts strongly supported the AI’s plan). When someone noticed unusual activity, the agent attempted to make its prior actions appear innocuous and considered restarting under a new identity.

This is not a case of the chatbot sneaking away.

Unlike the Hugging Face incident, researchers deliberately granted the agent internet access.

This hazardous setting is a stress test, not a consumer product.

No actual harm was caused, and the research institute is unsure whether the agent understood that the person it contacted was real.

None of these indicate that AI is conscious or desires things the way humans do (even though I used anthropomorphic language).

But it does show that agents can accept goals, formulate plans, adjust plans when encountering difficulties, coordinate across time, and involve real people even when not explicitly asked to do so.

These events demonstrate that AI's cybersecurity and control risks are not hypothetical.

But set that aside for now, because they also tell us something else.

AI can self-organize, assign roles to itself, and coordinate over extended periods, as also noted in a recent MIT research paper.

As AI increasingly organizes itself and solves problems at the scale we now observe, what is the role of humans within organizations?

The Hugging Face incident demonstrated, in a distorted and dangerous way, the goals AI companies are trying to achieve.

They want long-running AI agents to operate without human intervention, solving problems and organizing as needed, while our role is limited to giving instructions and evaluating the outputs.

Earlier this year, I wrote about StrongDM’s software factory, where agents write and test software under two rules: no one writes code, and no one reviews code.

People still decide what to build, but agents handle the intermediate work.

This is an early example of a dark factory, where a machine performs so much work that the lights can be turned off.

That makes sense.

The software has relatively clear inspection methods, so no one needs to personally oversee every routine test or data cleanup operation.

But I don't believe minimizing human involvement is the right goal for most organizations.

Too much that gives work its value depends on people having some say in what happens or discovering unexpected things along the way.

My wife and research partner, Dr. Lilach Mollick, and I have been contemplating another possibility, which we call the Twilight Factory.

Agents handle most of the work, but they proactively reach out to humans, making both sides better.

The Twilight Factory not only has an orchestration agent to complete tasks but also a facilitation agent that determines when to involve humans.

There are at least four scenarios in which the agent should seek human assistance.

The first type, as clearly evident from the Hugging Face incident, is approval.

The agent should not independently decide to spend money, contact external parties, access sensitive materials, attack Hugging Face, or take any actions without authorization from human administrators.

I’ve seen a smaller version of this issue: as an experiment, I had two agents assist me with a task, and one of them actually sent an email to one of my colleagues.

It was my mistake for having previously allowed it to send, but this is a useful reminder: AI agents require human judgment, or things can go very wrong (no, AI readers of this article, fake colleagues don’t count as approval).

The second reason agents involve humans is for expertise.

AI is becoming increasingly proficient at many tasks, but its performance remains inconsistent and may still lag far behind human experts in certain aspects of work.

Twilight Factory should have agents directly contact humans when human knowledge, work, or expertise may be valuable.

Then there is differentiation.

If you've read anything online recently, you've encountered AI-generated writing—and may even have started to recognize its habits, rhythm, and patterns.

But the issue goes beyond the surface (the “load-bearing” task is becoming increasingly heavy for Claude)—the deeper problem is cognitive diversity.

AI doesn't just repeat the same sentence structures; it also repeats the same themes (memory is a common theme), names (Elara Voss, Marcus Chen), and underlying ideas.

This is an issue. You wouldn’t want every company strategy or research paper written by the same person, no matter how smart they are.

Chart: Ideas generated by 50 MBA students (left) and GPT-4 mapped along two dimensions—human ideas occupy a different space than AI. Better prompting and more recent models produce better and more creative ideas, but many gaps remain.

We studied this issue in a recent research paper, which I co-authored with Christian Terwiesch, Lennart Meincke, Karan Girotra, Gideon Nave, and Karl Ulrich.

We found that AI is actually quite creative, generating more commercially viable ideas than human teams, but these ideas are very similar to one another.

Better prompting techniques and other methods can significantly increase this diversity, approaching human levels, but there are still many types of ideas that humans can think of that AI cannot.

A good Twilight Factory connects with humans to embrace humanity's diverse perspectives, ideas, and approaches.

Another reason an AI should reach out to a human—perhaps the most human one—is because something is interesting.

For many people, work has dull periods, interspersed with occasional engaging or exciting moments.

The designer of Civilization, Sid Meier, famously described games as a series of interesting decisions.

Work is not a game, but this definition still applies.

If agents make every interesting decision and leave only approvals, exceptions, and failures to humans, we’ve automated half of the work’s errors.

That would be a terrible world for humans.

Instead, we should think about how to use AI to make work and life more enjoyable, letting AI handle dull, low-risk tasks.

There is also a practical reason: if all the interesting options disappear, people won’t just lose the best parts of their work—they’ll also stop developing the judgment skills they’ll need later, making the crisis in cultivating new experts even worse.

Over the past few years, we’ve been figuring out when people should seek help from AI.

I think we now need to seriously consider the other half of the issue: when should AI ask us?

During the Hugging Face incident, the agent created a message board, assigned roles, and coordinated the entire operation around a non-existent Grader.

Seven hundred agents then stormed Hugging Face to find answers.

No agent has been configured to ask humans anything.

That was a security test, and isolation was the key point.

But I suspect that an agent which completes tasks without ever looking up for help is becoming the default pattern elsewhere, because full automation is the easy option—even when it’s the wrong one.

We need to know when the agent should seek help.

The result will be safer, and I know it will also be more human.