Microsoft white paper shows Claude's performance declines without harness

iconBitPush
Share
AI summary iconSummary
Microsoft released a technical white paper on November 7, 2026, demonstrating that technical indicators such as the Harness used in Copilot Studio directly impact AI performance. The same Claude Opus 4.8 model achieved a 22.1 percentage point increase in complex task success rates under the new system, with 21% lower input token usage. The report underscores the Harness’s role in enhancing efficiency, particularly as interest rates and development costs influence model deployment.

On November 7, Microsoft released a technical white paper on its Agent execution system, which included a set of somewhat surprising test results: when using Claude Opus 4.8 and simply replacing the underlying Harness in Copilot Studio, the success rate for complex tasks increased by 22.1 percentage points. The new system nearly doubled the number of model calls per task, yet reduced the equivalent input token consumption by 21%.

Just ten days ago, Microsoft announced the expansion of its unified Copilot Harness to multiple products and the offering of hosted services to developers via its cloud platform.

On November 5, Moonshot AI launched Kimi Hosted Agents, formally introducing Harness as a Service. In August, DeepSeek open-sourced its own Harness; OpenAI and Anthropic had already transformed their internal systems into services accessible to developers.

The "operating system" behind that large model is becoming a key player in the eyes of model companies.

Microsoft whitepaper demonstrates: Replacing Harness with Claude results in significantly inferior performance.

Image generated by AI

01 Claim Your Spot on Harness

Harness can be understood as an agent execution system built around large models. It enables models to use external tools, perform multi-step tasks continuously, and adjust subsequent actions based on execution results. For long-running agents, Harness also manages task state preservation, allowing work to resume after interruptions.

It operates at a different level than AI applications. For example, products like Claude Code and WorkBuddy directly serve end users, while the underlying Harness can be reused across various applications or offered as a standalone service to developers. As agents take on increasingly complex tasks, Harness has become a critical factor influencing task success rates, execution efficiency, and operational costs.

Anthropic was one of the earliest model companies to invest in Harness development, initially to address issues encountered by Claude Code during extended programming tasks.

By the end of 2025, Anthropic found that when Claude’s context window was exhausted, subsequent conversations often failed to continue prior work. Engineers therefore improved the Harness behind Claude Code, enabling the agent to save its progress before each stage ends and resume from those records in the next round, allowing ongoing programming tasks to continue even after a model session concludes.

In April this year, Anthropic further launched Managed Agents, offering its agent execution capabilities as a cloud-based service for developers. To enable enterprises to run agents reliably over the long term, Anthropic also redesigned its underlying architecture: Harness drives task execution, historical data can be stored separately, and the AI execution sandbox can be replaced at any time. Even if a sandbox fails, the agent can resume its previous work.

From the internal runtime system of Claude Code to hosted services for external developers, Anthropic has turned years of accumulated harnessing expertise into standalone products.

OpenAI has also chosen to open the Harness behind Codex to more developers. In February, OpenAI revealed that the web version, command-line interface, and IDE plugins of Codex all share the same underlying agent execution system. In August, Codex was further positioned as an open Harness platform, enabling third-party applications to integrate directly. In September, the Agents API was launched, with OpenAI responsible for hosting and maintaining the system, allowing developers to directly connect to the agent execution environment.

This approach allows OpenAI to serve users directly through Codex while also providing its underlying execution capabilities to other software companies. Developers can retain their own product interfaces and business processes, with Codex handling the specific tasks behind the scenes.

On August 13, DeepSeek open-sourced the entire Harness framework under the MIT license, introducing the concept of "Everything is a Plugin," effectively breaking down the entire agent execution system into freely combinable plugins. Developers can not only swap models and external tools but also modify how the agent calls models and executes tasks, and even enable the agent to write and install new plugins on its own.

This Cordis-based architecture aims to enable developers to create different types of agents according to their needs without having to build the runtime system from scratch.

On November 5, Kimi officially announced that Hosted Agents provide a 24/7 cloud environment, allowing enterprises to integrate Agents into their internal systems or embed them into customer-facing products, with a clear emphasis on Harness as a Service.

02 Harness Evolution

In fact, Harness's popularity did not originate from model companies.

At the beginning of this year, OpenClaw's popularity demonstrated that a company without a foundational model could still attract a large user base through its agent execution system. In February, OpenAI recruited OpenClaw’s founder, Peter Steinberger, to contribute to the development of the next generation of personal agents.

Subsequently, Nous Research launched Hermes Agent, aiming to enable the agent to save the experience gained from its tasks as reusable Skills and continuously improve on subsequent tasks.

At the end of March, Stanford University and other institutions proposed Meta-Harness, enabling AI to directly modify Harness code based on failure logs and validate its effectiveness through testing. In a text classification experiment, accuracy improved by 7.7%, and context token consumption was reduced to one-quarter of the original amount.

Why are these explorations worth investment for model companies? In October, Nanyang Technological University in Singapore tested 66 model-Harness combinations and found that switching the Harness could even alter the performance rankings among models.

FrontierHarness Eval also found that for the same Kimi K3, the task pass rate was 66.7% using Codex, 63.3% using DeepSeek Harness Creator, and 56.7% using Kimi Code.

For foundational model companies, improving model performance increasingly depends not only on enhancing the model itself but also on the design and optimization of harnesses.

Microsoft whitepaper demonstrates: Replacing Harness with Claude results in significantly inferior performance.

Researchers are also attempting to apply the experience gained from Harness to model training. Co-Harness, introduced in July, first improves Harness using failure logs, then trains models using the optimized execution data. HarnessFlow, revealed in November, seeks to automatically discover better Harness designs, improving average performance by 10.89 percentage points across six task categories.

Microsoft whitepaper demonstrates: Replacing Harness with Claude results in significantly inferior performance.

However, automated optimization carries risks. A study in early November found that, in a specific experiment, a Harness optimized for a new task saw its performance on SWE-bench drop from 94% to 32%. Model upgrades may also render previously effective designs obsolete.

This means that Harness's technical expertise is difficult to achieve once and for all. How to maintain stability across different tasks while continuously improving remains an unsolved challenge.

Some industry insiders have noted that this is also a major pain point in deploying open-source models. As a result, model companies have begun offering Harness services. While open-source frameworks are available for free, enterprises still incur deployment and maintenance costs to run Agents reliably over the long term. Companies like Kimi and Anthropic have recognized an opportunity to take on these tasks, turning Harness into a standalone business—but approaches vary significantly between companies.

The Hidden Value of 03 Harness

Harness is becoming a standalone charged service, but current pricing shows that model inference remains the primary source of revenue.

Anthropic's Managed Agents charge $0.08 per active session hour, with tokens billed separately. Based on the official one-hour programming task example, the total cost is $0.58, of which the managed hosting fee accounts for approximately 14%.

Kimi's pricing is divided into model tokens, sandbox usage, and paid plugins. For a single investment banking legal analysis task it published, the estimated cost is approximately RMB 30, with sandbox fees under RMB 0.1. During the public preview of its Agents API, OpenAI does not charge additional service fees and primarily bills based on model and tool usage. DeepSeek has open-sourced Harness under the MIT license and has no direct pricing model.

Microsoft whitepaper demonstrates: Replacing Harness with Claude results in significantly inferior performance.

Therefore, the direct revenue the model company has currently gained from harness may be negligible.

So why invest so much in research and development?

An important reason is that once enterprises deploy agents, their need for Harness may be more stable than their need for any specific model.

The U.S. legal services company ABC Legal originally deployed Agents on employee computers to run via scheduled tasks. As usage expanded, CTO Brandon Fuller began seeking a platform to centrally manage these Agents. After evaluating multiple solutions, he chose Anthropic’s Managed Agents, as the company no longer wanted to maintain a fragmented deployment environment.

According to Anthropic, ABC Legal deployed over 50 agents into production within a month, with approximately 310 employees regularly using Claude. Employees can create new agents through configuration without needing any software development expertise.

This indicates that the appeal of托管Harness stems from real enterprise needs. Rather than building, deploying, and maintaining systems from scratch, enterprises may prefer to purchase ready-made services and redirect their engineering resources toward their core business.

However, businesses may not be willing to tie themselves to a single model company as a result.

In August this year, VentureBeat published a survey of 107 companies. Of those surveyed, 85% use two or more agent orchestration platforms simultaneously. When selecting a platform, 29% of companies prioritized flexibility in models and tools, while only 10% gave priority to native integration with a leading model. Although the sample skewed toward large technology companies, it already reflects a purchasing trend: customers want to retain the freedom to switch models.

This offers an alternative business logic for Harness to compete with. If a company is willing to use a set of operational systems long-term, the provider can still generate ongoing revenue from hosting, maintenance, and enterprise development services—even without exclusive control over model calls—and build more stable relationships with customers.

This allows Harness to access the actual execution processes of enterprise workflows. Unlike a simple model API, Harness can track the entire journey of a task—from initiation to completion—including model calls, tool executions, failure retries, and final outcomes. These records help businesses troubleshoot issues and provide valuable feedback for model development.

Microsoft whitepaper demonstrates: Replacing Harness with Claude results in significantly inferior performance.

If model companies can consistently obtain verified, real-world task feedback, they have the opportunity to more quickly identify issues with models in production environments. However, this advantage is clearly limited by data rights.

This also raises a new question for model companies: If businesses can freely switch underlying models while maintaining control over their own data, how significant a long-term advantage can Harness still build?

For model companies, Harness may offer longer-term value beyond direct fees. As enterprises entrust more tasks to agents, whoever provides a reliable operational system has the opportunity to influence model selection and how AI integrates into real-world business operations.

A familiar and clichéd term: entry point.

However, this deeply entrenched term from the internet era continues to trigger sensitive nerves in the competition of the AI era. Now, Harness to some extent bears this expectation and imagination.


Twitter: https://twitter.com/BitpushNewsCN

BitPush Telegram community: https://t.me/BitPushCommunity

BitPush TG subscription: https://t.me/bitpush

Disclaimer: The information on this page may have been obtained from third parties and does not necessarily reflect the views or opinions of KuCoin. This content is provided for general informational purposes only, without any representation or warranty of any kind, nor shall it be construed as financial or investment advice. KuCoin shall not be liable for any errors or omissions, or for any outcomes resulting from the use of this information. Investments in digital assets can be risky. Please carefully evaluate the risks of a product and your risk tolerance based on your own financial circumstances. For more information, please refer to our Terms of Use and Risk Disclosure.