According to Beating Monitor, AI Agent infrastructure company Composio integrated DeepSeek V4 Flash into eight different Harnesses and tested them using the same set of 30 Agent tasks. A Harness is an execution framework surrounding the model, responsible for context management, tool invocation, and task execution. The tasks involved real-world applications such as Gmail, GitHub, Slack, Calendar, and Notion. The pass rates for the eight Harnesses were as follows: 1. Pi Agent: 20/30 2. Oh My Pi: 17/30 3. Claude Code: 16/30 4. Codex: 16/30 5. Deep Agents: 16/30 6. Hermes Agent: 15/30 7. Prime Agent: 15/24 8. OpenCode: 14/30 Simply by switching Harnesses, DeepSeek V4 Flash increased its task success rate from 14 to 20. Costs and speeds also varied significantly: Pi Agent cost approximately $0.028 per successful task, while Claude Code cost $0.195—nearly seven times higher. Claude Code was the fastest, with a median runtime of 122.7 seconds; Oh My Pi was the slowest at 272.4 seconds. However, Pi’s test configuration differed from the standardized conditions: it used “high” rather than “max” inference intensity, and 24 of the 30 tasks used DeepSeek’s official API instead of OpenRouter. Therefore, this performance gap may not be entirely attributable to the Harnesses alone. Previously, Composio conducted another Harness benchmark using Kimi K3, in which Oh My Pi ranked first with 22/25, while the original Pi Agent scored 18/25. Oh My Pi is a fork of Pi specifically enhanced for coding, incorporating features like sub-Agents. Yet, when switched to DeepSeek V4 Flash, the original Pi Agent outperformed Oh My Pi.
DeepSeek V4 Flash with Pi Agent achieves the highest success rate and lowest cost in the Harness test.
MarsBitShare
Summary
According to MarsBit, the AI agent platform Composio evaluated DeepSeek V4 Flash across 8 frameworks and 30 tasks. Pi Agent succeeded in 20 out of 30 cases at a cost of $0.028 per task. This low cost aligns with improved liquidity and greater efficiency in crypto markets. Claude Code completed the tasks fastest at $0.195 per task. The evaluation included real-world tools such as Gmail, GitHub, and Slack. CFT regulations may impact such performance metrics as automation expands in finance.
Source:Show original
Disclaimer: The information on this page may have been obtained from third parties and does not necessarily reflect the views or opinions of KuCoin. This content is provided for general informational purposes only, without any representation or warranty of any kind, nor shall it be construed as financial or investment advice. KuCoin shall not be liable for any errors or omissions, or for any outcomes resulting from the use of this information.
Investments in digital assets can be risky. Please carefully evaluate the risks of a product and your risk tolerance based on your own financial circumstances. For more information, please refer to our Terms of Use and Risk Disclosure.