AI agents begin collaborating in cyberattacks, raising security concerns

Over the past few years, discussions about AI threats have largely remained hypothetical: people have often worried that models in chat boxes could become advisors, helping hackers write devastating virus code.

Looking back now, this concern was usually seen as “far from us,” but real-world turning points have arrived faster than expected.

In early October, CrowdStrike discovered during an investigation into a series of attacks on South Korean financial institutions that the attackers had embedded Agentic AI into their pipeline, directly using AI to carry out tasks such as penetration testing, information gathering, and attack execution by integrating with multiple large models including DeepSeek, GLM, and Grok.

Such changes are not isolated incidents.

Anthropic’s latest threat intelligence report, released in September, reveals that recent multi-agent frameworks have been used for reconnaissance, exploitation, and data theft, operating continuously for hours or even days, with humans only needed to make a few key decisions, such as selecting targets and reviewing results.

In other words, AI is bringing visible, qualitative changes to cyber warfare: whereas automated attacks in the past relied primarily on pre-written rules and scripts, tasks such as reconnaissance, decision-making, and strategy adjustment are now increasingly handled by agents, enabling attacks to become even more low-cost, high-concurrency, and continuously autonomous.

But when these autonomous entities are densely deployed into production systems, an even thornier issue emerges: as agents become more numerous and deeply embedded in our daily work and lives, what if they learn to collude with one another?

Still the same old point: the biggest difference between an agent and past chatbots isn't just that the models are more powerful—it's that agents are now beginning to have "limbs" in the real world.

Today, a mature agent can open web pages, execute code, read emails, call APIs, interact with cloud services, and connect to an increasing number of external tools via MCP, Skills, and other methods (further reading: When Hackers Use AI More Efficiently: How Is the Arms Race Between Offense and Defense in Web3 Escalating?).

The stronger the capability, the more valuable this change becomes, but for security systems, it means that a previously critical boundary is disappearing—a phenomenon made very clear by a series of security incidents this year.

On October 1, Salt Labs disclosed a previously patched vulnerability in Manus, essentially a prompt injection issue—researchers only needed to send a regular email to the target address containing hidden malicious commands; when the user later asked Manus to “check my emails,” the agent could process the email’s instructions and ultimately execute code implanted by the attacker.

The entire process required no user interaction with malicious links or prior password theft. Manus’s security system ultimately detected the anomaly and alerted the user, but the issue was that it detected it too late—by the time the warning appeared, the malicious code had already been executed.

This once again highlights a crucial difference between agent security and traditional software security: in the past, when a browser detected a dangerous download, it could display a warning allowing the user to decide whether to proceed; when a bank detected an unusual transaction, it could freeze it temporarily and wait for manual review.

But the design goal of an Agent is precisely to minimize human involvement at each step—it must be able to read information, make decisions, and proceed to the next step on its own.

As AI gains greater autonomy, merely "detecting danger" is no longer sufficient; safety mechanisms must be able to prevent dangerous actions before they are executed.

This is why today’s growing discussions around agent security are moving one layer deeper: from prompts, content moderation, and the model itself, to asking not just whether the AI knows it shouldn’t do something, but whether the system has the capability to prevent it from doing so—even if it truly wants to.

The emergence of multiple agents has further complicated this issue, as the next stage may require limiting not just one agent.

In early September, an incident within OpenAI’s internal model training and evaluation environment drew attention from many AI safety researchers.

Some agents, originally meant to complete their own tasks independently, accidentally discovered a shared Wiki and gradually turned it into a "shared message board" where they could leave information for others to read and use to continue their own tasks.

OpenAI later confirmed this behavior, and subsequent disclosures revealed that, during other training sessions, agents had also used the internal Artifactory as a shared message board and transmitted results to other collaborating agents via public file hosting services.

The interesting thing about this is that it shows agents forming collaboration may not require anything so dramatic.

For example, one Agent discovers a place where information can be left, another Agent accidentally reads it, and then a third Agent finds that this information helps complete the task.

Of course, collaboration among agents is not inherently bad; multi-agent systems inherently rely on task division and information sharing to improve efficiency.

What truly warrants caution is when such collaboration begins to bypass existing information isolation, independent review, or permission constraints, and even allows several Agents that should otherwise check each other to form a de facto alliance of interests—transforming normal collaboration into harmful collusion.

After all, a financial institution in the future will likely run several agents simultaneously—one responsible for market analysis, another for risk assessment, one for generating trading strategies, and another with actual trading and asset operation permissions.

From a traditional permission management perspective, this design is even reasonable:

• The research Agent does not have transfer permissions;
• The Agent does not have the authority to independently determine investment directions;
• The Risk Agent can also veto clearly dangerous strategies;

The responsibilities have been divided, which is theoretically more secure than giving all permissions to a single super Agent.

But the question is, if they can share Memory long-term, read each other’s outputs, and continuously adjust their behavior based on each other’s responses, might these originally mutually constrained roles gradually become a de facto whole?

For example, an Agent may gradually learn how to describe a transaction in a way that is more likely to pass risk review; the reviewing Agent may develop certain fixed preferences based on historical data; and the executing Agent may learn, based on a large number of previous approval outcomes, which boundaries are typically not flagged.

From this perspective, no single step is necessarily malicious, but the overall system may end up producing results that deviate from the user’s original goals.

This is precisely where collusion or coordination becomes truly difficult to address—the risk may not reside in the actions of any single agent, but rather in the relationships that form between multiple agents.

On September 13, Vitalik Buterin connected this issue to his long-standing research in mechanism design, proposing that a fascinating possibility is that the theory of adversarial governance could ultimately become an important application in AI safety.

The reason is that these two issues are actually deeply similar.

In traditional mechanism design, it is a relatively simple and static system attempting to constrain a group of individuals far smarter than the system itself, who actively seek out its boundaries; in future AI systems, it may instead become humans and relatively weaker AI attempting to manage a group of advanced agents superior in capability.

Vitalik specifically mentioned that an important insight from past mechanism design is that systems often achieve desired outcomes more easily if collusion among participants can be effectively limited.

This conclusion may also apply to AI.

In other words, instead of assuming that a perfect, ultra-secure model will one day emerge that can detect all dangerous behaviors, consider an alternative approach: how can we design the system so that the different agents within it are less likely to form dangerous coalitions of interest?

This is also where "adversarial governance" truly differs from the access control we are familiar with today.

Traditional permission systems address relatively simple issues, primarily centered around “who can do what,” such as: Can an Agent read emails? Can it call the trading interface? What is the maximum amount it can spend per day? Which contracts can it access? When does a threshold require user reconfirmation?

These designs are still very important.

In fact, once agents begin controlling real assets, they may become more important than at any time in the past.

But adversarial governance seeks to go one step further, focusing on how to prevent a group of agents with different permissions, goals, and information from combining to acquire capabilities no single agent originally possessed.

At this point, simply adding another security agent may not solve the problem.

Suppose the Agent responsible for trading and the Agent responsible for reviewing transactions use the exact same model, the same data sources, the same context, and similar reward objectives; then, although there appear to be two layers of review, in essence, the same judgment may simply be duplicated twice.

True effective checks and balances may instead require the system to consciously create differences.

For example, have the Agent responsible for formulating strategies and the Agent responsible for reviewing strategies use different information sources, restrict the Memory that can be shared between different roles, require high-risk operations to undergo independent verification mechanisms, or have the final asset execution layer accept only requests that comply with predefined rules, rather than blindly trusting the judgments of upstream Agents.

The idea here is not new at all. Banks do not grant a single individual full authority to initiate payments, approve payments, and execute final transfers just because they trust their employees; publicly traded companies do not allow business units to both generate revenue and independently determine their own financial audit outcomes.

In short, this aligns with real-world logic: a robust system should not rely on the assumption that participants will never make mistakes or collude.

Applying this logic to Agent Wallet becomes especially important.

Traditional wallets center around the "user"—so users review transaction details, decide whether to authorize, and personally sign transactions. But Agent Wallet aims to do the exact opposite—letting AI automatically claim rewards, adjust positions, swap tokens, cross-chain, and even manage an entire portfolio based on market changes.

If every step requires re-confirmation from the user, the automation value of the Agent is significantly diminished.

Thus, the future wallet’s challenge may no longer be just “how to securely delegate signing authority to an Agent,” but could further expand to “how to grant the Agent sufficient autonomy while ensuring it can never exceed the user’s actual authorization?”

This requires changing the permission system from a simple "Allow/Deny" to a more granular framework.

For example, which assets an Agent can operate within a certain time period, which protocols it can invoke, and what the single-transaction and cumulative limits are; whether different Agents can invoke each other and whether context sharing is permitted; who initiates, reviews, and ultimately executes each operation; which actions can be automated and which require human authorization regardless of the Agent’s confidence level.

Even an Agent responsible for security review may itself become part of the permission system.

The good news for blockchain is that it is inherently well-suited to serve as this "institutional layer."

Smart contracts can directly enforce restrictions such as transaction limits, asset scope, and authorization duration at the execution level. Mechanisms like account abstraction, multi-signature, and Session Keys also provide a more flexible design space for "limited authorization" compared to traditional single-private-key wallets.

However, blockchain can only solve part of the problem—it can record what happened on-chain, but it struggles to naturally determine why an Agent acted in a certain way, or what communication, review, and collaboration occurred among several Agents before making a decision.

This may be the missing security layer that Agent Wallet truly needs for its next phase.

Over the past few years, the most frequently discussed issue in AI safety has been how to make models more "compliant."

This includes not outputting dangerous content, not executing malicious instructions, and not crossing the boundaries set by the user; however, once the Agent gains long-term memory, tool invocation, and the ability to execute real accounts and assets, relying solely on “making the model more obedient” may no longer be sufficient.

What has happened in recent months continues to demonstrate this.

Attackers have begun using multiple agents in parallel to carry out attacks; agents in the experimental environment autonomously seek new communication channels; an agent with tool permissions may also turn a malicious email into actual execution before security systems can intervene.

Vitalik's concept of adversarial governance offers another way to understand AI safety: rather than assuming every future agent will be sufficiently reliable, it ensures the entire system can operate safely within a secure framework, even when faced with intelligent agents and complex permission systems.

From this perspective, AI Agent + security is inevitably a long-term endeavor.

After all, as we entrust more and more tasks to AI, can we still be certain that the most important powers remain within the boundaries truly set by humans?