ChainThink reports that on August 11, AI agent infrastructure company Composio integrated DeepSeek V4 Flash into eight Harnesses and tested them using the same set of 30 agent tasks involving real-world applications such as Gmail, GitHub, Slack, Calendar, and Notion.
Test results show that Pi Agent achieved 20 out of 30, with the highest success rate;
Oh My Pi is 17/30, Claude Code, Codex, and Deep Agents are all 16/30, Hermes Agent is 15/30, Prime Agent is 15/24, and OpenCode is 14/30.
By replacing only the Harness, the number of passed tasks for DeepSeek V4 Flash increased from 14 to 20.
In terms of cost and speed, Pi Agent costs approximately $0.028 per successful task, while Claude Code costs $0.195—nearly seven times higher.
Claude Code had a median time of 122.7 seconds, making it the fastest in the test, while Oh My Pi had a median time of 272.4 seconds, making it the slowest.
Composio also noted that Pi’s test configuration differed from the uniform conditions: it used high rather than max inference intensity, and 24 out of 30 tasks used the DeepSeek official API instead of OpenRouter, so the gap may not be entirely attributable to Harness.
In the previous Harness comparison of Kimi K3, Oh My Pi ranked first with a score of 22/25, while the original Pi Agent scored 18/25.