Although outside attention on Monday focused more on Anthropic’s latest financial data disclosed in its revised S-1A filing—revenue surging, with expenses rising just as sharply—the AI lab also dropped another “little bomb” in the details of the filing.
This document appears to warn that the company is nearing the creation of a model capable of recursive self-improvement (RSI).
RSI refers to the point at which AI becomes strong enough at training itself to trigger a dramatic acceleration in its capabilities. For AI doomsayers, RSI represents an irreversible catastrophe: AI builds itself into something powerful enough to escape human control and potentially disregard the harm caused in pursuit of its goals.
For AI optimists, RSI—or some reliable AI version capable of improving itself—represents the promise of this technology: it will become a tool that delivers unimaginable gains in productivity, scientific discovery, and more.
Anthropic appears to view RSI as both an opportunity and a risk. In its S-1 filing, the company states that it uses AI in its research and development, while also warning: “Using AI to help build stronger AI may also introduce significant safety risks, particularly when it substantially accelerates progress. This is one of the areas covered by our Responsible Scaling Policy, which outlines our plans for monitoring and mitigating risks as we approach what is known as ‘recursive self-improvement.’”
When the term RSI gained popularity earlier this year, many believed it was merely a buzzword, or at best an overly complex, difficult to achieve, define, or prove goal.
However, on Monday, Anthropic added new language to its “Risk Factors” section stating that: 1) the company had used AI to develop several new models without realizing the magnitude of their capability leap at the time; and 2) the company may soon reach its internal definition of RSI (below, as emphasized in the original):
In addition, our increasing use of AI systems to accelerate our own research and development may lead to our models' capabilities improving faster than our ability to develop, validate, and implement corresponding safety, alignment, and security measures. As a result, we may have failed in the past to recognize, and may fail in the future to recognize, significant acceleration in research and development until after it has already occurred.
For example, our internal estimates and expectations for future models, rather than assessments of current models, suggest we may soon reach the definition of “significant acceleration enabled by AI” in the Responsible Scaling Policy. We have not yet classified any current model as exceeding this threshold, and we are still evaluating the underlying data.
However, Anthropic also stated in its documentation that none of its models have currently crossed this RSI threshold, and the company has committed to continuing human involvement in its development process for the foreseeable future. The company said: “We anticipate that it will not be safe to use Claude for fully autonomous design and construction of successive generations of frontier models without meaningful human oversight or understanding—even if and when this becomes technically feasible. Therefore, we are considering updating our Responsible Scaling Policy to maintain human control over the development of Anthropic’s frontier models.”
Skeptics may point out that the S-1 is both a disclosure document and a marketing tool, and claiming to be close to a powerful model capable of self-building may fall into the latter category.
However, there is no doubt that Claude’s capabilities, particularly in programming, mathematics, and cybersecurity, have made tremendous strides this year. If Anthropic believes that the likelihood of RSI is now close at hand, we should perhaps take this seriously and act accordingly—possibly even implementing appropriate governance.