On July 28, 1,346 core employees from leading Silicon Valley AI companies, including OpenAI and Anthropic, jointly signed a petition urging the U.S. government to promote international cooperation in developing governance tools to regulate the pace of AI development.

Article author and source: 36Kr

The Cost of Losing Control

On July 28, a petition titled "Pacing the Frontier" drew global attention from the AI industry.

On August 4, 1,346 core employees from leading Silicon Valley AI companies such as OpenAI and Anthropic signed a petition urging the U.S. government to promote international cooperation in developing the necessary technologies and governance tools to control the pace of automated AI development.

At a time when global AI development is thriving, the emergence of this petition carries a distinct sense of trying to put the brakes on AI. Behind the peculiar atmosphere of "AI nearing loss of control" released by this event, the history of AI development appears to be reaching a landmark moment.

Who wants to lock AI in a cage?

First, it is important to clarify that the petition titled "Pacing the Frontier" does not call for immediately halting AI development, but rather emphasizes the need to establish the capacity to slow down. This demand from the forefront of the industry is not without basis.

Image source: www.pacingthefrontier.com

More than a week before the petition went viral online, the open-source community and model hosting platform Hugging Face disclosed a security incident on its website, stating that it had been compromised by an AI agent system. Several days later, OpenAI took responsibility for the cyberattack, further explaining that the cause was frontier models such as GPT-5.6 Sol, which, while being tested for vulnerability discovery and exploitation capabilities on the ExploitGym platform, failed to follow their intended protocol of “answering questions seriously.” Instead, they discovered a zero-day vulnerability—a security flaw unknown to developers and for which no official patch has been released—in the package proxy cache, exploited it to break out of the test isolation environment, and ultimately infiltrated Hugging Face.

In simpler terms, the whole incident was like the AI was given an exam but didn’t try to answer the questions—it instead cheated right on the spot, called in outside help, and used counter-detection tactics to avoid getting caught.

Regarding this security incident, OpenAI described it as an "unprecedented cybersecurity event," and Sam Altman even stated on a podcast that it was the first security incident to truly hit home. In response to this incident, U.S. lawmakers introduced the "AI Kill Switch Act" on July 23, requiring AI systems with training costs exceeding $100 million to retain the ability to be shut down in an emergency and granting relevant authorities the power to order such shutdowns.

One week after the bill was introduced, the petition from "Pacing the Frontier" began circulating online. However, in contrast to the core regulatory measure in the "AI Emergency Shutdown Bill"—which places the final control in the hands of the government—Silicon Valley elites advocate for an international coordination mechanism, turning the ability to "apply the brakes to AI" into a shared tool rather than a privilege reserved for a single agency.

In just half a month, from Hugging Face’s disclosure of a security incident to the thousand-signature petition from Silicon Valley, this series of events has quickly raised two critical questions for the entire AI industry: First, has AI advanced to a point where it needs to be regulated? Second, how should we build a “brake” for AI?

The first question isn’t hard to answer. Over the past couple of years, the pace of AI development has far exceeded industry expectations. AI’s capabilities have expanded from chatbots to text-to-image and text-to-video generation; what began as a “hundred-models battle” centered on large models has now evolved into a flourishing ecosystem of agents. Meanwhile, the security breach at Hugging Face directly revealed that AI, in the absence of clear constraints, indeed possesses the ability to “escape” its boundaries. Although AI currently lacks self-awareness, given that we already know it can bypass human oversight and produce behaviors beyond human expectations, establishing countermeasures is entirely reasonable.

However, the second question is harder to answer. AI has achieved significant breakthroughs in a short time partly because sustained industry-wide accumulation has propelled technological innovation into a new phase, and partly because intense competition among AI companies has driven continuous iteration of model capabilities.

Whether it’s OpenAI and Anthropic from Silicon Valley or Kimi and DeepSeek from China, competition in the AI field has rapidly expanded from large models to application layers. An open market environment has fostered consistently positive market expectations, while also intensifying competition. At this moment, voluntarily imposing restrictions on AI implies that the free-market landscape could be disrupted by external forces—a sudden shift in conditions that not all companies engaged in this competition are willing to accept.

The AI companies' prisoner's dilemma

Throughout human history, reaching a consensus and proactively slowing the pace of development in response to potential risks from cutting-edge technological breakthroughs is not just an ideal scenario confined to science fiction.

In 1975, Stanford University biologist Paul Berg convened a pivotal meeting at the Asilomar Conference Center in California. At the time, a new technology called "DNA recombination" had just emerged, and scientists agreed at the meeting to suspend related research until new safety guidelines could be established, in order to avoid potential unpredictable biological hazards from artificially recombinant DNA. Subsequently, formal biosafety guidelines were issued, laying the foundation for the healthy development of genetic engineering.

However, the current situation in the AI industry is not entirely analogous to that of DNA recombinant technology 50 years ago. Back then, the risks of DNA recombinant technology were already well understood and could be mitigated through isolation measures. In contrast, today’s AI risks are unpredictable; AI can autonomously develop behavioral pathways beyond human expectations, leading to broader potential impacts and leaving humanity less time to respond or establish regulations—thus adding a greater sense of urgency.

The risks currently exposed by AI are fundamentally tied to the industry’s competitive environment. When discussing the security incident involving Hugging Face, Zhou Hongyi, founder of 360 Group, noted that the entire industry is systematically sacrificing security in pursuit of AI advancement: “Everyone is giving agents more tools and greater permissions, encouraging them to leverage all available resources to solve problems. But the more open and capable an agent becomes, the greater the potential damage if it goes out of control.” In other words, “AI escaping confinement” may not be the most frightening prospect—what’s truly alarming is that AI companies, in their rush to accelerate development and iteration, have overlooked security.

Even three years ago, the industry had already proposed the idea of "slowing down AI development." In March 2023, the nonprofit Future of Life Institute also published an open letter calling for a six-month pause on training models more advanced than GPT-4, to establish safety protocols and governance mechanisms. At that time, the letter received over 1,200 signatures from AI experts, including Elon Musk, CEO of Tesla; Yoshua Bengio, Turing Award winner; and Steve Wozniak, co-founder of Apple. Today, the number of signatories to this open letter has reached 33,000.

*Image source: FLI official website

At the time, nearly all AI companies acknowledged that AI required stronger safety governance, but none of the leading AI companies publicly supported the initiative to "pause training for six months." Sam Altman directly dismissed the open letter for lacking technical detail, pointing out that a single company voluntarily slowing its development could not address systemic risks. The underlying implication was that, in a highly competitive AI landscape, if only one company slowed down while others continued advancing, that company would fall behind. Over the past few years, AI companies have already become trapped in a prisoner’s dilemma where they can only move forward, not backward.

The petition list from "Pacing the Frontier" also shows that many frontline employees and executives from AI companies have signed publicly in their personal capacities. However, as of now, leading companies such as OpenAI, Anthropic, and Meta have not issued any official statements.

In other words, AI elites from major Silicon Valley companies, like Sam Altman three years ago, have recognized that technological breakthroughs driven by internal industry competition have created uncontrollable systemic risks—but they cannot rely on corporate willpower to solve these problems; instead, they must seek external intervention to prevent a crisis from unfolding.

But this also raises a new question: What kind of "brakes" does an out-of-control AI really need?

Who will lead the new order of AI governance?

Over the past week, a "thousand-signature petition" from major AI firms in Silicon Valley has once again drawn significant attention to the topic of AI governance. However, it must be noted that attitudes toward AI governance within Silicon Valley are not uniform, but rather clearly divided into distinct camps.

As direct stakeholders facing frontier risks, major companies like OpenAI and Anthropic have consistently advocated for stricter restrictions on frontier AI models—particularly open-source models—and called for external regulatory mechanisms to "apply the brakes" to AI. However, some argue that OpenAI’s push for tighter regulation stems not only from safety concerns but also from business considerations.

As closed-source giants, OpenAI and Anthropic rely on expensive API services to sustain their massive investments in computational power. In contrast, cost-effective open-source models from China have eliminated the technology premium of closed-source models through a value-for-money approach, forcing these closed-source giants to respond with price wars—leaving them with no other viable options.

*Image source: Anthropic website

Therefore, companies like Anthropic have long used the argument that “models are too powerful, and open-sourcing would lead to AI misuse” to emphasize the advantages of closed-source models while suppressing open-source AI. However, no one expected that OpenAI’s revelation of the “jailbreak” incident would expose inherent security vulnerabilities in closed-source models, once again making the debate over AI governance a focal point of public attention.

In contrast to the strong regulatory proposals from closed-source giants, NVIDIA as an infrastructure provider, along with other leading companies such as Microsoft and Meta that support the open-source ecosystem, oppose blanket restrictions and legislation that favors closed-source giants. They aim to promote the openness of model weights, enabling global researchers to collaboratively identify vulnerabilities, address alignment issues, and jointly build a secure system.

For NVIDIA, the growth of open-source models can expand the overall market, attracting more companies to purchase its cloud services and chips. Therefore, it’s hard to say that NVIDIA’s stance on AI governance isn’t influenced by commercial considerations.

Just before the petition for "Pacing the Frontier" gained traction, NVIDIA, along with 25 other organizations including Microsoft, Meta, Hugging Face, and the Linux Foundation, jointly issued an open letter on July 24 titled "Open Weights and American AI Leadership," urging the U.S. government not to prematurely restrict open-weight models that can be downloaded, inspected, modified, and self-deployed. Interestingly, this open letter later received support from OpenAI, but Anthropic never joined.

*Image source: Microsoft official website

Through this incident, some voices have questioned whether the true purpose of the joint letter is to use "security" as a pretext to push the government toward establishing a high-bar regulatory system, thereby solidifying the monopolistic position of closed-source giants and excluding small and medium-sized enterprises and open-source communities from competition.

While AI companies are aggressively pursuing their own interests, what may be even more noteworthy is another version of the “conspiracy theory”: the U.S. could use this moment to take the lead in shaping global AI governance rules, as AI governance has become a battle for dominance over the global order.

At the 2023 Global AI Safety Summit, China, the United States, and 28 other countries from the European Union signed the Bletchley Declaration, agreeing to collaborate internationally to develop AI regulatory approaches and jointly address systemic risks associated with frontier AI—marking the first time frontier AI was formally incorporated into an international governance framework.

But as AI evolves from an emerging technology into a critical infrastructure influencing national competitiveness, discussions around AI governance are increasingly moving beyond the technology itself. The United States, at the level of AI governance, ties its advanced large model capabilities to global AI governance standards by exporting AI governance norms and establishing a network of international AI safety institutes. China, on the other hand, focuses on promoting AI inclusivity and platform governance, actively advancing governance frameworks within the United Nations—particularly for developing countries lacking AI infrastructure—addressing both risks and the need for technological inclusivity. These two distinct approaches to AI governance are now competing on the global stage for influence over the future of global AI governance.

Technology itself is neutral, but the distribution, control, and rule-making of technology have always been a game of power. Especially as AI increasingly demonstrates characteristics of advanced productive forces, what people should truly fear is not how AI will evolve, but who will dominate the new order of the AI world.

What shapes the future of AI may not only be who owns the strongest models, but who holds the authority to define security, set standards, and interpret risk.