Multiple chip manufacturers are increasingly exploring new evaluation methods, driven by the fact that agents are transforming CPU workloads.Article author and source: Wall Street Journal
The application scenarios for agents are becoming increasingly specialized.
From AI programming tools like Claude Code to office assistants capable of processing documents and operating software, and further to personal assistants that help users plan trips and manage schedules, agents are taking on an increasing number of specific tasks.
The expanding range of application scenarios has also placed new demands on underlying computing power.
Against this backdrop, Arm recently introduced a CPU performance evaluation methodology tailored for agents, aiming to help businesses assess how different CPUs perform in agent tasks through tests that more closely reflect real-world scenarios.
This is not an isolated case. AMD previously proposed a CPU workload evaluation framework for Agent AI, incorporating testing of tool execution, data retrieval, and task orchestration; Intel has also conducted related tests on Xeon 6 processors, with a focus on throughput and concurrent execution efficiency for Agent tasks.
Multiple chip manufacturers are increasingly exploring new evaluation methods, driven by the fact that agents are transforming CPU workloads.
Over the past few years, the competition for computing power in large models has primarily centered around GPUs, with metrics such as token generation speed and throughput becoming key indicators of AI inference performance.
As the Agent begins executing actual tasks, in addition to invoking large models for reasoning, it also needs to access databases, execute code, and call tools—all of which may consume significant CPU resources.
Ravi Malhotra, Director of Architecture for Arm’s Cloud AI Solutions, used a software development agent as an example, noting that the model first performs inference and generates code, after which the CPU still needs to handle code building, compilation, and testing—operations that can consume significant time.
“In this type of use case, it’s critical that the CPU completes its portion of the work as quickly as possible, so control can be returned promptly to the more expensive GPU or accelerator to execute the next step in the pipeline,” Ravi Malhotra told outlets such as AllThingsD. “If the CPU isn’t fast enough, the more expensive GPU will sit idle.”
More complexly, the performance bottlenecks of different agents vary. Some tasks are primarily limited by GPU inference speed, while others are constrained by CPU sandbox execution capability or data retrieval efficiency. Even for the same type of agent, the performance bottleneck may change depending on the model, deployment environment, and specific task.
For example, in software development, code compilation and testing often require continuous CPU resources, making single-agent execution performance more critical. In contrast, scenarios such as enterprise customer support or personal assistants typically involve short individual operations but may require handling numerous requests simultaneously, placing higher demands on CPU concurrency.
These differences have also led CPU performance evaluations to focus on the efficiency of executing complete agent tasks and performance under different workloads.
Arm proposes combining component-level testing with end-to-end agent benchmarking.
Among these, component-level testing covers components such as the web server, database, model service, and sandbox execution, while end-to-end testing incorporates Agent benchmarks like SWE-bench and Terminal-Bench to evaluate performance across the complete workflow.
Beyond traditional metrics such as single-core performance and multi-core throughput, Arm also focuses on the number of concurrent agents, task execution efficiency, and performance per rack.
Ravi Malhotra emphasized that performance evaluation requires balancing throughput with SLA (Service Level Agreement) requirements, rather than simply pursuing the ability to run more agents simultaneously.
Due to varying requirements across different scenarios, Arm does not recommend a uniform weight allocation for all Agent use cases; instead, it advises enterprises to select appropriate evaluation metrics based on their specific applications.
As agents gradually integrate into enterprise business processes, these performance metrics become important references for enterprises when selecting and upgrading CPU infrastructure.
Ravi Malhotra believes that scenarios such as software development, customer support, supply chain management, and personal assistants will become key drivers for enterprises to upgrade their CPU infrastructure.
“For enterprises, scenarios such as customer support, supply chain management, and personal assistants place greater emphasis on performance per rack, where the scalability and energy efficiency advantages of the Arm AGI CPU are particularly pronounced,” said Ravi Malhotra.
Ravi Malhotra further noted that whether upgrading existing infrastructure for Agent applications or designing and developing entirely new infrastructure, these scenarios will create clear upgrade demands and are expected to be key drivers of growth and adoption for Arm AGI CPUs.
For businesses, the value of improved CPU performance must ultimately be reflected in actual operational efficiency and cost savings.
The workload varies significantly across different agents, and whether the performance advantages observed in benchmark tests can truly translate into improved task execution efficiency and reduced deployment costs still requires further validation through real-world applications.
Risk Disclaimer and Liability Waiver
The market carries risks; invest with caution. This article does not constitute personal investment advice and has not taken into account individual users’ specific investment objectives, financial situations, or needs. Users should consider whether any opinions, views, or conclusions presented herein are suitable for their particular circumstances. Investment decisions based on this content are made at your own risk.