DevOps Pioneer Warns: Adopting AI Agents Requires System-Level Changes, Not Just Tools

Stop fixing the code produced by the Agent; fix the system that generates the code.

Article author and source: InfoQ

If a developer simply can't use an Agent effectively, the issue may not lie with the developer, but rather with the company having never established a functional system for the Agent.

Many companies' so-called AI transformation still stops at purchasing tools like Cursor and Claude Code for developers, conducting a few training sessions, and then leaving employees to figure things out on their own. If the agents ultimately underperform, the blame falls back on the users.

However, Patrick Debois, the person who coined the term DevOps, believes: "Developers need to make a crucial mindset shift: when an agent fails to complete a task as expected, don't just modify the code it generated—instead, improve the entire system, not just the prompt."

In Debois’s view, this is an inevitable shift as software engineering moves from deterministic systems to non-deterministic, probabilistic systems and workflows. It involves not only technical changes but also transforms how developers, teams, and entire organizations operate. However, this transformation cannot be achieved by a single engineer or confined to a single team—it, like DevOps, can only be truly realized when scaled across the organization.

The core issue is not whether developers will use agents, but whether the company can reorganize its teams, platform, and collaboration methods around agents.

The key points are as follows:

• Stop fixing the code produced by the Agent; fix the system that generates the code.
• If anyone on your team is still using a "YOLO" approach to vibe coding, you should stop them immediately. Engineering practices are essential not only for maintaining your system but also for ensuring the continuous improvement of the Agent itself.
• A dim factory may not be fully dark, but instead retains a bit of light, meaning you must decide how much risk to take on for each function—not all functions are suitable for full autonomy.
• The person you're looking for combines exceptional AI skills, strong engineering foundations, and a willingness to share and collaborate.
• Your moat is capturing the accumulated knowledge—the business context you're now embedding into your skills, context, and even Harness constraints.

In 2009, a lot of people told me that the idea of continuous delivery was insane.

Translator’s note: In 2009, the industry commonly adopted a model of large-scale, infrequent releases occurring monthly or less often. It was widely assumed that the more frequent the releases, the higher the risk. Additionally, strict silos existed between development and operations, containerization and cloud-based automated infrastructure had not yet matured, and standardized CI/CD pipelines were lacking. Traditional testing and change approval processes aimed to eliminate as many defects as possible before deployment. In contrast, continuous delivery introduced the revolutionary concept of frequent, incremental, and随时可发布的 releases, challenging conventional beliefs about software deployment risk and process control. As a result, to most companies at the time, this approach seemed almost fantastical and wildly impractical.

Now, the dark factory is encountering exactly the same resistance.

Translator’s note: The dark factory refers to an AI-driven autonomous software production model in which humans input only specifications, and AI autonomously handles coding, testing, and deployment without requiring human review of each line of code—differing from traditional software factories that still rely heavily on engineer involvement throughout the process.

I keep hearing the same phrase in various settings: “This doesn’t work here.” But what it really means isn’t that the technology doesn’t work—it’s that “we’re not ready.” It’s not that they don’t want to implement it; it’s that the current structure of the organization can’t support this model.

Many people are currently discussing how to optimize agents using loops and how to set up harnesses—these are all great. But what I want to say is that we will all eventually reach that level of technical capability; one day, these will become standardized commodities, even packaged as services by leading labs. When that happens, there will be no technical barrier left. The real differentiation will lie in how your organization restructures collaboration around this technology.

So I assume we’re all heading toward dark factories. What I’ve observed at Tessl and other companies is that when people start adopting these technologies, the dynamics of collaboration change completely. If you’re familiar with Conway’s Law, you know there’s a reciprocal relationship between how an organization is structured and the tools it creates—how you organize people determines the kind of systems you build. But today, I’m not here to talk about how to make your agents better; I’m here to discuss how this transforms your team dynamics, your platform, and your entire organization.

I assume most of you here work as part of a team rather than alone—team collaboration is entirely different from typing away at Claude Code by yourself.

Now everyone likes to say that developers will ultimately become conductors, orchestrators of agents. I think this statement is spot-on—it’s exactly the path we’re on. We’re increasingly becoming managers of agents, needing to navigate our relationships with them.

But the problem is, I’ve heard many developers say in private: “We didn’t get into this field to do this—we never imagined we’d spend so much time optimizing prompts or writing better specs. We’re engineers; we work with technology. This creates an identity conflict, and we constantly ask ourselves: Is this really the role I want?”

Later, the concept of "context engineering" emerged, offering developers a bit of relief. It suggests that this isn't just about crafting prompts—you also need to test, evaluate, deploy, and optimize them, which does carry a certain engineering flavor. But honestly, many developers still feel hollow working solely with prompts and specs, as if they've transitioned from engineers to "prompt administrators."

But in practice, I observed an interesting turning point: when we began introducing Harness, loops, and moving the entire organization toward greater autonomy, an entirely new technical pathway opened up. Suddenly, developers needed to build tools for the Agent—and this instantly reignited a group of people. Developers who previously thought, “This isn’t my job,” suddenly became energized. They said, “Yes, we can do this! We have this knowledge! We can use programming to make this system better.” So it’s fascinating: while we had been constantly talking about “abstraction, abstraction, more abstraction,” the sense of craftsmanship reemerged in another space, creating new room for more hardcore engineering work.

People often ask me: How do you handle those who are skeptical? My answer is always this: These people are actually your treasures. They possess vast amounts of implicit knowledge and judgment that you need to feed into your Agent. You can tell them: “Please bring out all your knowledge and scrutiny”—this will make your Agent and Harness better. If you encounter someone who resists and constantly complains, “The code this generates is terrible,” treat them as fuel—channel that anger and skepticism into driving improvements to your system.

Let me offer a suggestion to the company’s developers: make a major mindset shift—stop fixing the code produced by the Agent, and instead fix the system that generates that code. As someone said a few years ago: “Don’t build the thing; build the thing that builds the thing.” We are now at this level of abstraction, crafting “the thing that builds things” through Context, Harnesses, and loops. Those still stuck at the level of “Human in the Loop,” auto-completion, or tuning prompts need to think about elevating themselves to a systems-thinking perspective.

What we truly need to do is minimize human intervention through solid engineering practices. At first, everyone enjoyed “vibe coding”—throwing out a prompt, getting a result, and moving on without concern. But it’s now increasingly clear that when we give prompts to agents, we’re not just issuing commands; we’re saying: write tests along with the code, update documentation, and follow coding standards. The very things we used to tell good engineers, we’re now telling agents directly. If anyone on your team is still using a “YOLO (just make it work)” approach to vibe coding, you should stop them immediately. Engineering practices aren’t just critical for maintaining your system—they’re essential for helping your agents continuously improve.

I’m beginning to see a new ritual emerge in some of the more advanced teams: they still hold planning and retrospective meetings, but the focus of the discussions has completely shifted. Instead of asking, “What went wrong with the code?” they now ask, “What went wrong with the system?”

I also noticed an interesting division during the planning meeting. Tasks that were clearly defined and sufficiently scoped could be directly handed off to the Agent, as Harness has improved significantly and can now handle such well-defined tasks. Meanwhile, tasks with ambiguous boundaries or those requiring discussion are still kept for humans. As a result, a natural division emerged during the meeting: these cards go straight through the Agent pipeline, while those we’ll discuss together.

Developers typically go through a learning cycle: first learning Prompt, then better SPECs, followed by Context, Harness, and iteration—the entire industry is climbing this same cycle. What team leads can do is set the pace and constraints for this process, such as telling their team: “Stop tweaking the Prompt; focus on making the Context reusable.” “Great, that step is done—let’s move on to the next one.” The value of a team lead lies in establishing this rhythm; simply saying “figure it out on your own” won’t work.

There’s also a cascading effect: once your team’s productivity starts to surge, downstream teams—such as those handling GTM (Go-to-Market)—may struggle to keep up, and even users might fall behind. That’s why you need automation to support them; your framework shouldn’t stop at coding—it must extend to their workflows. The same principle applies to upstream demand inputs: if requirements don’t come in fast enough, your team will hit a bottleneck, and these stages must also be integrated into this new workflow.

There are countless metrics out there today—things like token spending, and so on. But I’m increasingly convinced that only two metrics truly measure productivity. The first: count how many manual interventions are still required to get an agent to do something correctly. This number should continuously decrease. The better your harness, the clearer your context, and the more precise your guidelines, the lower this number becomes. The second metric is the multiplier effect that emerges when you shift from individual work to a shared system: fix something once, and everyone benefits. It’s not about one person becoming ten times more efficient; it’s about a single optimization to the agent system creating a multiplier effect across all users.

You can start by launching it within a single repository or a small team, sharing context and collaboratively improving Harness. But what you truly want to achieve is extending this effect across the entire organization—at which point, we must discuss platform teams.

The platform team is a typical shared-services organization, and currently they may be focused on infrastructure, cloud services, MCP gateways, and similar areas, with less attention paid to Agents. However, a number of new initiatives are emerging that require their involvement—such as a skills registry (to prevent everyone from independently inventing the same skills in their own corners), a context evaluation system (to determine whether a given context is useful and whether its value can be quantified), and dedicated guardrails and identity management for coding agents (with questions like: Under whose identity does the Agent submit code? What are the boundaries of its permissions?). The platform team needs support to help them evolve into this new central role.

This is difficult—you need a clear owner to drive it. But who should that be? The platform team? The developer experience team? The former typically doesn’t touch development-level concerns, while the latter rarely touches infrastructure. So you need some kind of fusion, but this fusion won’t happen automatically. You must ensure there’s a dedicated person accountable for driving this centralized effort; otherwise, your teams will just work in silos, and no “paved road” will emerge.

Why does each of our teams have to invent its own authentication system integration method? This is a shared component and should be placed in the registry. Why are we all building our own harnesses? If we all use the same linter and the same set of security scanning tools, these become reusable components. I believe this will follow the same path as the early adoption of cloud infrastructure—gradually consolidating into the platform registry.

But the problem is, if anyone can just dump anything into this central repository, it will quickly become a sprawl. For example, if someone uploads a skill, who maintains it? Then another person forks a similar skill—so which one should I choose? Therefore, someone must clearly own each domain and ensure that the component is testable and modular, allowing others to extend the Context or the security scanning portion within the Harness. You need to approach this in a centralized way, rather than letting things circulate randomly within the organization.

Reaching consensus is hard. It’s not as famous as the tabs versus spaces debate, but sometimes it feels just as intense. If you try to get two development teams to agree on how to work, it requires a lot of communication and mediation. In the end, you’re likely to end up not with one paved path, but three or four, from which they can choose. If they insist on creating their own system, that’s fine—but it’s on their own budget. The centrally maintained option is the “easy path,” designed to attract everyone to use it.

If people blindly use these shared capabilities, you must make them see the cost. As soon as you make the spending visible, they will naturally want to optimize. It’s the platform team’s responsibility to make costs transparent: How much was spent? How much value was delivered? If I can reduce the number of Agent iterations, that’s optimization. But if I can’t see this metric—only the final outcome—I have no way to act. Visualization is the foundation of all optimization.

So my core argument is: we must move from individual developers working in isolation to team-level shared context and shared components, and ultimately to an organization-wide “multiplayer system.” That’s where the multiplier effect will explode, because you’ll have a flywheel where improvements can radiate in multiple directions simultaneously.

One level up, how does the VP of Engineering think about this? I can pretty much predict the story that will unfold in your organization: a hackathon or lunch-and-learn, sharing success stories, creating a shared Slack channel, launching a champions program. These are all generic transformation tactics—Agile transformation did this, DevOps did this too, nothing new here.

On the other hand, we also know that the strategy of “issuing licenses, providing training, letting everyone go wild, and letting a thousand flowers bloom” has never succeeded. The result of a thousand flowers is usually a thousand weeds—many blooms, but none that bear fruit. Therefore, I advocate that the organization clearly empower team leads and platform teams to take ownership of this. It cannot be accomplished by a single super-individual; someone must be formally authorized to drive it forward.

Finding someone to help can also be frustrating. Current job titles are a mess—AI Product Engineer, Forward Deployed Engineer, Agentic Engineer, AI Engineer… these terms don’t really mean anything substantial. You can’t judge a person’s level of experience by their title, because the entire industry is still immature. However, when posting job requirements, these terms do send certain signals and attract candidates with specific intentions—but they don’t guarantee the candidate actually possesses the corresponding skills. I’ve even heard more outrageous stories, like candidates using AI to whisper real-time answers into their ears during interviews; whenever the interviewer asks a question, suggestions from the AI come through their AirPods.

So I’ve noticed more and more companies adopting this interview approach. First, give them a problem to solve using AI—encourage them to use it fully. If AI helps them succeed, it demonstrates their ability to leverage AI effectively. Second, ask them to review their own solution and explain: “Why did you choose this approach? How did you verify it’s correct?” Here, you’re testing their testing skills and engineering judgment. The first part assesses AI utilization; the second evaluates engineering fundamentals. Third, observe how they collaborate—whether they’re willing to share ideas, and whether they’re open and collaborative or prefer working alone. Some individuals have strong technical skills but hoard everything themselves; in the Agent era, these people can become bottlenecks.

The person you’re looking for is someone who can leverage AI to its fullest, has solid engineering skills, and is willing to share and collaborate—combining all three. It’s not someone who has merely studied ML or AI, nor a decoding expert, but rather a hybrid. You may not find someone who excels in all three areas, and that’s okay—perhaps a candidate is exceptionally strong in one area but needs guidance in another. Also, don’t lump these skills together under labels like “junior” or “senior”; they are distinct dimensions—one person might have “senior” AI utilization skills but “junior” willingness to collaborate.

The VP of Engineering still needs to deliver results to upper management. We’ve purchased so many licenses—can we prove the return on investment? Has delivery sped up? There may be promises, but it’s hard to prove. Has quality improved? Again, it’s difficult to say. But going back to the two metrics I mentioned earlier, you can show how many fewer interventions occurred, how much improvement was achieved, and how much the reuse rate increased. This is far easier and more compelling than trying to compare coding productivity “with vs. without an Agent.”

So, when someone complains that the Agent is too costly and suggests limiting its budget, your instinct shouldn’t be “cut all spending,” but rather “how can we optimize spending?” The simplest approach is choosing the right model—not every task requires the most powerful one; some can be handled just fine with a more affordable model. Educate developers on which models to use in which scenarios, and further equip them with better context and harnesses—this will help the Agent avoid unnecessary detours and drive costs down even further.

There’s also the topic of team size. Having a single all-in-one person handle everything is the ultimate dream. But if you do the math carefully: that person typically needs complementary skills, such as a product manager or designer. Then you also need to account for backup personnel—what if someone goes on vacation? That brings you back to three people. Then there’s the need for someone to oversee production and tickets; if you’re extremely efficient, the same people might take on those roles part-time. But once you start fixing bugs, your feature development speed slows down. And don’t forget new hires—you need to guide them and show them what “good” looks like. So I still believe it’s simply not feasible within an organization to reduce every team to just one or two people.

Finally, a dark factory may not be entirely dark but could retain some dim light (a dim factory), meaning you must decide how much risk to accept for each feature—not all functions are suitable for full autonomy. You can invest more in auditing, such as provenance: who made the code change? Was it a human or an agent? Add validators to verify whether the code is truly functional, and invest in situational awareness when automated processes fail. This spans a spectrum from complete micro-management (every line of code reviewed by a human) to full autonomous approval (assuming all agent outputs are correct). Your task is to select the appropriate level of automation for different types of changes based on their risk profile.

And I believe your moat lies in capturing the accumulated knowledge—the business context you’ve embedded into your skills, your context, and even your Harness constraints. To me, this transforms continuous delivery into continuous learning. Ask yourself: How quickly can we swap a new thing into the system and remove an old one? That’s your responsiveness. If you can continuously improve this capability, the critical question shifts from “How do I make the entire system more reliable?” to “Can I maintain reliability while changing more and more parts of the system?”

If you take away only one sentence, it should be: The winners won't be the lone super players, but those who understand how to improve organizations across multiple levels.