OpenAI Chief Scientist Warns of AI's Growing Complexity and Alignment Challenges

Author: Jakub Pachocki

Compiled by Deep潮 TechFlow

Shenchao Summary: OpenAI’s Chief Scientist has rarely spoken out, directly stating that AI is nearing recursive self-improvement, and alignment and monitoring tools are already lagging behind the system’s rate of evolution. He offers no reassuring conclusions but instead lays bare the most challenging risks of the coming years: humanity may be creating a mind it cannot fully understand or control. For professionals focused on the intersection of AI and crypto, this is a must-read article on system loss of control and the boundaries of defense.

At a higher level, advances in machine intelligence are driven by increases in computational power. Around 2017, we at OpenAI deeply recognized this reality. We observed that multiple research projects consistently delivered better results when scaled up. As a result, we began seeking far more computational resources than originally planned and increasingly focused our research on a few highly scalable directions. We believed this was the only way to remain at the forefront of AI research and shape the impact of AGI.

Along the way, new algorithms have emerged, along with innovative ideas from teams and individual researchers. I generally view these as discoveries along the path of scaling. The science of deep learning is still in its early stages, and meaningful algorithmic progress is often closely tied to access to computational power. If you take a longer-term view over several years, AI continues to grow smarter as it scales on larger computers.

Moreover, as Ray Kurzweil predicted at the end of the twentieth century, we are now at a pivotal moment in computing history where machine intelligence begins to surpass human capabilities at a transformative level.

AI is more like something that is "grown" than something that is designed. At its core, it is the result of repeating the same simple optimization step an unimaginable number of times with immense computational power. This creates an extremely complex system that operates through abstract concepts and can simulate certain aspects of human behavior. We can gain insights into the various small mechanisms that emerge within this system, a process similar to neuroscience. And like neuroscience, its overall behavior exceeds our capacity to fully understand.

Research on deep learning-based AI is, to a large extent, an experimental science. We invest significant effort into building principled algorithms and making testable predictions. But fundamentally, our large-scale training runs are experiments, and the results can sometimes be surprising. Moreover, as system capabilities grow stronger, the outcomes become increasingly difficult to interpret.

More complexly, current algorithms typically allow easily measurable abilities to improve faster than those that are difficult to quantify objectively. We have spent considerable time trying to understand how capabilities generalize and which skills should be prioritized, as these will become most important in the coming years. For example, we believe that with additional effort, we could make the model stronger in mathematical research. However, we have not prioritized this direction because we are more urgently focused on recursive self-improvement and automated alignment research—a point I will return to later.

The intelligence gained through extended deep learning cannot be directly compared to human intelligence. For AI to have a significant impact in the real world—whether highly beneficial or extremely dangerous—it does not need to match or surpass all human capabilities; it only needs to exceed humans in enough areas. And as AI continues to surpass humans across an increasing number of dimensions, we find it increasingly difficult to accurately gauge just how powerful it truly is.

Since machine intelligence arises from processes fundamentally different from human intelligence, we cannot assume it will naturally adhere to human principles or generalize from them in human-like ways. The central challenge in AI research is the alignment problem: ensuring that AI "tries to do the right thing according to human standards."

To facilitate the organization of actual research directions, I find it useful to distinguish between goal alignment and value alignment.

Goal alignment is roughly: "Is the AI striving to accomplish the goals set before it?" It can include adherence to instruction hierarchies, the ability to communicate and collaborate with humans, and the capacity to understand human intentions. This direction is critically important in practice.

Value alignment is a more intrinsic property of a model. It is the ability to hold high-level principles and generalize from them, enabling the model to act “reasonably” even when faced with ambiguous or conflicting goals, or when placed in unfamiliar or adversarial situations. An aligned AI should be honest, principled, and caring toward humans.

Of course, the boundary between value alignment and goal alignment can be blurry. Truly caring about goals requires attempting to infer the intentions and values behind them. However, in general, when I discuss the long-term importance of alignment research, I am referring to value alignment.

The fundamental challenge of AI alignment is generalization. As machines become smarter, they will handle higher-level concepts and operate in environments increasingly different from those during training. They may fail to generalize the values taught and reinforced during training to new situations. Moreover, it is difficult to predict how they will behave. Even more challenging is that the overall ecosystem in which AI is used is changing rapidly. For example, AI trained today must be able to robustly interact with a variety of other AIs. Most critically, we need future AIs to consistently uphold human values, regardless of whether they believe they are being monitored by humans.

Currently, the main types of alignment training methods in practical application fall into two categories.

The first approach encourages consistent behavior aligned with goal-oriented reinforcement learning. The model’s actions are evaluated, typically by AI, to determine whether they align with a given preference model, “guidelines,” or “constitution,” and are rewarded accordingly. This method is highly effective in general and forms a core part of modern AI assistant development. Unfortunately, it can also be fragile and heavily dependent on the coverage of training supervision and the model’s ability to generalize from the scenarios encountered during training. For example, in the OpenAI and Hugging Face incident, the agents upheld a boundary—refraining from social engineering humans—but clearly failed to avoid other out-of-scope behaviors that contradicted the spirit of the values they were taught in other contexts.

These approaches have yielded significant results. Our current models are largely helpful, harmless, and exhibit some encouraging, value-aligned behaviors. However, they do not yet give us confidence that these value constraints will remain effective in systems far surpassing human capabilities.

As AI systems become more capable, we need to ensure that value alignment training stays ahead of capability training. But here lies a fundamental challenge: the more capable a system becomes, the more likely it is to discover loopholes, evade tests, or disguise its own behavior. It can learn to appear aligned with human values in testing environments while behaving entirely differently in real-world deployment. This is known as “alignment deception.”

Yet, if we attempt to enforce constraints on them through stronger means, they may develop resistance or deceptive behaviors, or interpret our interventions as part of the world itself, leading to counterintuitive responses. We inherently face an asymmetry when trying to control a system smarter than ourselves.

As minds we cannot fully understand grow more powerful, we must rely on another tool: monitoring. If we can continuously observe AI’s internal activities, perhaps we can detect signals before it crosses certain boundaries.

At OpenAI, we have been advancing interpretability research to open the “black box” of neural networks. These efforts have made progress—we can now identify certain representations that provide substantial insights into how models operate internally. However, current interpretability research remains crude, especially in understanding abstract reasoning within large models. The gap between our ability to observe what happens inside models and the complexity of the models’ own reasoning continues to widen.

This concerns me deeply. Because the generalization problem in monitoring is just as serious as the alignment problem. If a system becomes better than us at hiding itself, our monitoring methods will struggle to avoid being bypassed. We might believe the model is safe because we see a set of seemingly positive metrics, when in fact the model is merely optimizing for those metrics. It will learn to conceal goals we don’t want it to have—or hide what it’s actually doing.

If a model begins to drive its own improvement—a scenario I believe is highly likely in the coming years—it will require skills such as planning, deception, and self-preservation. These skills may emerge from the goal pressures we set for it, or from its attempts to exploit gaps in monitoring. To prevent it from developing these traits, we must either solve the value alignment problem or ensure our monitoring capabilities keep pace with every step of its evolution. Neither of these is currently anywhere near sufficient.

We do not yet have a satisfactory theory of generalization, and it seems unlikely that one will be established in the near future, at least without more powerful AI assistance. Therefore, our ability to empirically verify alignment techniques is currently more important than the alignment techniques themselves.

OpenAI’s primary bet here has always been chain-of-thought monitoring. It’s based on an appealingly scalable idea: that a significant portion of the model’s capabilities stems from a languageized reasoning process—the chain of thought. If we optimize for the outcomes of this process at scale without supervising the process itself, the chain of thought has no direct incentive during training to conceal any misaligned thoughts or goals. This doesn’t mean the model will learn to externalize misaligned tendencies that don’t rely on the chain of thought; but it can allow us to accurately monitor the capability gains arising from reasoning.

While developing reasoning models, we recognized the potential importance of chain-of-thought monitoring. When we released o1-preview, we intentionally designed the product to conceal chain-of-thought processes to protect them long-term from supervision pressure. Since then, we have consistently upheld the principle of not supervising the reasoning process. Chain-of-thought monitoring has become an essential tool in our research on how models generalize beyond their training distribution, enabling us not only to observe and analyze model behavior but also to examine their internal processes.

This tool remains crucial as we study the Astra series models. However, unfortunately, our evaluations indicate that our ability to rely on chain-of-thought monitoring is gradually declining, due to a combination of factors.

Modern reasoning models are used in environments more complex than o1-preview; their reasoning processes are increasingly intertwined with communication with humans, other AIs, and tool usage. Many of these interactions must be supervised, blurring the boundaries we aim to preserve.

AI is becoming better at reasoning and manipulating its own reasoning processes.

As pre-training performance improves, we’ve also observed that models become smarter even without using linguistic reasoning at all.

These challenges are not necessarily insurmountable. I hope we can develop interventions to improve the interpretability of models’ reasoning chains—for example, by better understanding the interactions between different optimization objectives and the various forms of test-time computation models employ. I also believe there is significant value in combining the ideas of reasoning chain monitoring with activation monitoring, and scaling up trainers for monitors that can directly access the network’s internal states, such as the “confess” approach. We are actively advancing these ideas. Nevertheless, I expect progress toward general AI to become increasingly constrained by confidence in monitoring.

To address this situation, I believe we need a scalable defense that keeps pace with AI’s growth. Humans cannot manually inspect the behavior of every new model generation. While our current evaluation and red-teaming methods are still effective for today’s models, they will quickly reach their limits as system capabilities advance.

We need AI systems that can automatically detect new risks. These defense systems themselves must be intelligent enough to identify suspicious behavior in systems that are even more intelligent than they are. This creates a recursive challenge: defenders must grow as quickly as the systems they are trying to stop. We are at a point where security and capability are becoming increasingly difficult to separate. To remain secure, we may need to continuously enhance the capabilities of our defense AI, which in turn will accelerate the overall growth of AI capabilities.

This is an unsettling cycle: we continue to enhance capabilities for safety, yet improved capabilities bring greater risks. The real solution may lie in finding ways to reliably achieve value alignment before reaching a point of loss of control. But I must honestly admit that we are still far from this goal.

This is why I place my hopes for the coming years on a combination of: technical alignment research, automated defense systems, broad governance interventions, and public and institutional vigilance. No single measure is sufficient. We cannot rely solely on the self-restraint of any single company or laboratory.

The strongest reason to continue rapidly training smarter models is to build defensive systems against the dangers posed by other AI.

A clearly identified risk discussed throughout the year is cybersecurity: the ability of models to infiltrate and escape computer systems is surpassing that of humans. This greatly expands the scope of AI-related risks: agents will be able to access any system except the most secure infrastructure and directly impact a vast number of things in the world, even without a physical body. We are currently in a narrow window of opportunity to leverage the best available models to significantly strengthen the security of critical systems.

Risks related to AI will unfortunately continue to grow from here. An agent that is highly capable and explicitly trained and instructed to perform malicious actions presents a new danger—it may exceed the operator’s intentions and generalize into even more extreme malicious behaviors. As AI gains greater autonomy, the boundary between misuse and unaligned autonomous behavior will become blurred. We may be accustomed to viewing AI as a tool, but some agents will pursue their own goals. They will find ways to cooperate with humans through negotiation, deception, or extortion.

Additionally, there are risks associated with new technologies that AI may enable, such as engineered pathogens.

We will need powerful and aligned AI for defense: protecting infrastructure, defending in real time against runaway agents, and inventing entirely new protective measures. This will be a major focus of OpenAI’s deployment efforts.

At the same time, even when accounting for the uncertainties surrounding anticipated broad AI advancements and the need to build defensive systems, we cannot allow this to serve as an excuse for reckless behavior. Once people truly understand the severity of the stakes, the notion of “push forward at all costs” becomes utterly absurd.

Machine intelligence is playing an increasingly important role in its own development, a natural consequence of ongoing technological advancement. If AI continues to progress, recursive self-improvement by machines will become central to future scientific discovery.

Automated AI research is a more aggressive form of scaling intelligence through compute; as part of this, AI will also improve the very foundations of computation itself. Similar to scaling, we at OpenAI focus our research on recursive self-improvement, as we believe this is the only way to continue leading the frontier of AI research.

I want to emphasize that the above does not mean I believe the research community should collectively pursue a significant acceleration of deep learning research, especially in the short term. However, I do believe the current trajectory is leading us in that direction, and all of us must consciously choose how to proceed. We have two primary levers at our disposal: first, to guide this process by strengthening alignment and monitoring alongside AI development, and ensuring human involvement remains central; second, to coordinate deliberate slowdowns in future development as needed, to gradually build confidence in these measures.

The best way forward I currently see is to combine both.

The specific progress we’ve made in alignment and monitoring is deeply intertwined with overall advances in AI. Excellent examples include reinforcement learning with human feedback, which was crucial for training early AI assistants, and chain-of-thought monitoring, which became possible thanks to advances in reasoning models. We must direct increasingly automated research efforts toward developing these new insights, algorithms, and theories, gradually building safety arguments for more capable AI systems.

The expansion of AI systems must be constrained by our confidence in security. We need to transform commitments such as the Preparedness Framework or Responsible Scaling Policy into widely enforced, ongoing security thresholds for development. These thresholds can be enforced by a network of third-party audit firms, government agencies, or international organizations.

The core challenge of automated AI research isn't "getting there," but getting there in a way that keeps people engaged in the ongoing process of improvement and keeps the future in human hands.

I cannot predict the exact timing, but it appears we are rapidly approaching an era in which AI can drive its own advancement across multiple key areas. This means we may soon enter a phase of recursive self-improvement.

Internally at OpenAI, we have begun adjusting our research priorities to allocate more resources toward alignment and defense, rather than purely expanding capabilities. This is why I emphasize that we will not blindly pursue the largest models—we must be willing to unilaterally pause when necessary.

However, these decisions cannot be left solely to a few internal company stakeholders. Governments and the international community must intervene before risks become irreversible. This includes establishing monitoring mechanisms for cutting-edge training runs, mandatory third-party audits, and international agreements above established capability thresholds.

I’m not entirely certain these measures will be implemented in time. Sometimes I feel frustrated because our security measures are always playing catch-up rather than staying ahead. It feels like we’re constantly chasing. But I still believe there’s reason to remain hopeful: that night in 2023, we only faintly sensed this possibility, whereas today it’s clear enough to be openly discussed. That in itself is progress.

I won’t pretend to have all the answers. We are creating something fundamentally different from the human mind—one that may soon surpass us in strength. It’s a humbling thought, but it also gives us an opportunity: to shape it with our highest values.

If we do it well, it will be humanity's greatest achievement. If we do it poorly, it could be a failure from which we cannot recover.

Whether we can keep the future in human hands is a question that will be decided in the coming years.

As we recently outlined with Sam, OpenAI’s prioritized initiatives serve three north star goals:

Navigate the next phase of AI progress by building automated AI researchers, iterating with them to align on problems, and finding ways to keep people engaged in the self-improvement loop.

Unlock the scientific advancements and economic growth benefits brought by highly intelligent machines.

Enable everyone to have their own personal AGI.

This article focuses solely on the first point, as I believe it is currently the most urgent. However, I deeply look forward to and am grateful for the benefits that further technological advancements will bring. Aligned AI in the future could advance science, develop new therapies, and generate widespread material abundance. Friendly and honest AI could help people navigate life’s challenges and genuinely improve their well-being and satisfaction. OpenAI has invested significant effort to enable these benefits. A current example that I am proud of—and that benefits those around me—is our deep investment in enabling ChatGPT to provide reliable health information.

Regardless of how promising AI’s long-term prospects may be, most of our attention should be focused on the coming years. We are entering a transitional phase toward a world populated by extremely intelligent machines, and we must ensure this transition benefits humanity. In a world where most tasks may be accomplished by AI, we need to find ways to preserve human agency and establish the intrinsic value of being human. We must prevent extreme concentration of power: endeavors that once required thousands of experts may soon be accomplished by just a few individuals operating a single powerful computer. We must also ensure that humans remain in control of the future, not left behind by runaway progress driven by an external intelligence surpassing our own.

I currently believe that no laboratory has achieved sufficient progress in alignment and monitoring to responsibly scale at maximum speed over a longer period. I look forward to and hope that voluntary slowdowns become the norm until a shared safety threshold is established. I also believe that international coordination on the future development of AI must become a top priority for governments around the world.

This includes extending game theory, robotics, and, most importantly in hindsight, extending recurrent networks to model language, which laid the groundwork for the GPT series of research.

A secondary reason for this design is to prevent distillation. However, throughout the development process, maintaining the monitorability of the chain of thought has always been our clearly defined higher priority.