Composio Tests Kimi K3 on Three Frameworks, Token Costs Vary by Up to 9x

According to Beating Monitor, AI agent infrastructure company Composio connected the same Kimi K3 model to Kimi Code, Hermes, and Claude Code to complete 28 identical tasks. The number of successful completions varied by only two across the three frameworks, but token usage and costs differed significantly. Kimi Code completed 22 tasks, Hermes completed 21, and Claude Code completed 20. The median token usage per task was 61,000, 67,000, and 340,000 respectively. Composio estimated the average cost per task at approximately $0.22, $0.28, and $2.00. Token usage for individual tasks varied by up to 30 times. Hermes was the fastest, with a median task completion time of 179 seconds; Kimi Code took 297 seconds, and Claude Code took 348 seconds. Another study by Writer yielded similar results: researchers replaced only the execution framework across 22 enterprise tasks and six models, achieving a 38% reduction in token usage, a 41% reduction in cost per task, a 44% reduction in time, while maintaining nearly identical task completion quality.