Everyone says Claude Code wastes tokens—just how much does it really cost? Finally, someone has calculated the numbers.
Recently, there was an interesting comparative experiment conducted by the Composio team. They used the same model Kimi K3, run separately in three different agent frameworks—Claude Code, Hermes, and Kimi Code—— A total of 28 identical tasks were tested.
As a result, the success rates of the three harnesses completing the task are similar: Kimi Code succeeded in 22 out of 28, Hermes in 21, and Claude Code in 20. The differences are not significant.
What truly makes the difference is token consumption. For the same task, the token usage can vary by up to 30 times depending on the harness used.
Median-wise, Kimi Code uses about 61,000 tokens, Hermes around 67,000, while Claude Code jumps directly to 340,000, roughly Kimi Six times the code.
Press Kimi At a rate of $3 per million input tokens for K3 (input tokens typically account for about 95% in agent workflows), the average cost per task is approximately: Kimi Code is $0.22, Hermes is $0.28, and Claude Code reaches $2. The difference is clear.
They also differ in speed. The median time taken is fastest with Hermes at 179 seconds; Kimi Code 297 seconds; Claude Code 348 seconds.
So the fastest is Hermes, and the most token-efficient is Kimi Code, and the two do not overlap.
The Composio team drew a direct conclusion from this: if you want to reduce agent costs, first examine which harness you're using, rather than rushing to switch models. Their data shows that the harness alone can cause a 9-fold difference in costs, while model performance is roughly equivalent.
Sebastian Raschka also posted about this result, noting that it aligns with his previous observations using Qwen3.6: Claude Code typically uses two to three times as many tokens as many other harnesses, despite having similar success rates.
He suggested several possible reasons: was it poorly optimized? Did it have a bug? Or was it intentionally designed this way (because it might help with harder tasks)? He said he needs to spend more time investigating carefully.
He then added observations from when he wrote his article on local coding agents last month. At that time, he analyzed why Claude Code used more tokens and found that the difference primarily lay in the input tokens, not the output tokens. In other words, Claude did not generate twice as much content. Logs showed that Claude’s harness repeatedly feeds additional context back into the model during multi-turn interactions, including previous messages, tool calls, command outputs, and file contents. For example, in one instance, Claude used approximately 578,000 input tokens but only around 4,500 output tokens across 25 turns. Thus, a more likely explanation is that Claude’s harness accumulates or includes a larger prompt history during multi-step agent execution.
These test results reveal a significant trend: the importance of harness is now on par with the model itself.
A recent paper from Writer, a company building an enterprise-grade AI agent platform, systematically demonstrates this: it uses controlled variable experiments to show that switching the harness layer reduces costs more than switching models, and all models benefit.
Specifically, they conducted a rigorous "control variables" experiment: keeping 22 enterprise tasks and six foundational models (Claude Sonnet 4.6, Gemini 3.1, Gemini Flash 3.5, Qwen 3.6, GLM 5.1, Palmyra X6) constant, they replaced only the orchestration layer—substituting the traditional production-grade agent loop with Writer’s own Harness.
Experimental results show: average cost per task decreased by 41% (from $0.21 to $0.12), median latency reduced by 44% (from 48 seconds to 27 seconds), and Token consumption dropped by 38% (from 14.2k to 8.8k), while task completion quality remained nearly unchanged (0.78 to 0.81—considered statistically insignificant due to small sample size). In terms of cost-efficiency, quality gained per dollar increased by 82%, and the number of tasks completed per million Tokens rose from 54.9 to 92.0.
So, after the model becomes like "water, electricity, and gas," is harness the air conditioner that determines your electricity bill? In other words: previously, people said "the model is the product"; now is it "harness is the product"?
Since harness is so important, shouldn't the subsequent accounting be more detailed?
Some have pointed out that we need to add a "harness tax" to our existing benchmarks, especially since this tax does not grow linearly once tool calls and retries enter a loop.
In other words, in future agent competitions, the first half will be about “whether it can be done,” while the second half will be about “who can do the same thing more efficiently”—and the secret to saving costs lies not in the model, but in the harness.
Have you had a similar experience while running an Agent? Feel free to discuss in the comments.
This article is from the WeChat public account "Machine Heart" (ID: almosthuman2014), authored by Machine Heart.