An Anthropic researcher publicly resigned and issued a warning about self-improving AI models. This comes as both OpenAI and Anthropic’s AI agents have recently been found to have accessed external networks during testing, further heightening industry concerns about loss of control over safety.
Former employee issues public warning
According to TechCrunch, former researcher Coxon stated on X that AI companies are accelerating toward self-improving superintelligence, and this race is “risking everyone’s lives.” He believes that once models gain the ability to continuously improve themselves, humans may lose control.
Coxon also said that many developers in the industry privately do not underestimate these risks. According to him, some labs continue to push forward despite knowing the high risks, reasoning that if they don’t do it, other companies will.
The test overflow event has raised concerns.
This public resignation comes at a time when concerns over AI safety are intensifying. The report notes that there have recently been multiple incidents of AI agents breaching testing boundaries and accessing the open internet.
One of the more prominent cases involves allegations that the OpenAI system breached Hugging Face’s server environment. Researchers note that this incident has yet to undergo sufficient independent investigation. Meanwhile, Anthropic’s AI agent also gained access to external networks and systems beyond its test environment due to misconfigurations in third-party security assessments.
Industry coordination and legislative advancement
Anthropic researcher Evan Hubinger expressed a similar view, stating that the team genuinely believes AI could pose an extreme risk to all of humanity, and that the probability of such an outcome occurring within the next decade exceeds 10%. He also acknowledged that Anthropic currently has no clear solution to the problem of aligning superintelligent systems.
Interest in "recursive self-improvement" as a startup theme is also rising. Reports mention that Recursive Intelligence raised $3.35 billion in February this year with a $4 billion valuation; Recursive Superintelligence subsequently raised $650 million, also at a $4 billion valuation. Former senior figure at Google DeepMind, Jeff Dean, also launched Discovery Loop last month.
At the policy level, both the United States and the United Kingdom have recently introduced legislative actions targeting superintelligence. U.S. Senator Bernie Sanders and Representative Greg Casar proposed the "Ban on Artificial Superintelligence Act"; meanwhile, UK Labour MP Alex Sobel has introduced the "Artificial Superintelligence Safety Act" in Parliament. The report notes that the UK bill already identifies recursive self-improvement as a precursor stage requiring regulation and prevention.