Independent evaluation agency Artificial Analysis has launched its first AI agent benchmark, AA-AgentPerf.

Author and source: AI New Era

With the same one megawatt of power, NVIDIA’s latest GB300 NVL72 can support 61,400 agents simultaneously, while the previous H200 can handle only about 2,600.

There is a full 20-fold difference in between.

NVIDIA's AA-AgentPerf results show that, under service standards of 20 and 60 tokens per second, the GB300 NVL72 achieves approximately 20 times the number of concurrent agents per megawatt compared to the H200.

On June 12, when NVIDIA first released these figures, the initial external reaction was that it was another display of performance prowess.

What has truly changed is not how powerful this generation of chips is, but the ruler used to measure computational power.

It is the new benchmark released by the independent evaluation agency Artificial Analysis: AA-AgentPerf.

Artificial Analysis calls it the industry's first reasoning benchmark specifically designed for AI agents.

Its primary metric is also different from before: not tokens per second, but "agents per megawatt."

In simple terms, it means how many agents the system can support simultaneously for every megawatt of power supplied.

FLOPS has been measured for years, and token throughput per second has been used effectively—why introduce the new benchmark AA-AgentPerf?

The old ruler can no longer measure the agent.

To answer this question, we first need to understand what kind of load an agent generates when running.

Artificial Analysis's assessment is clear: the most mainstream AI workloads in 2026 have nothing to do with what the old benchmarks were originally designed to measure—those benchmarks evaluated fixed-length synthetic requests while deliberately disabling optimizations that are actually enabled in production environments.

NVIDIA also provided an apt analogy:

A regular conversation is like a 100-meter sprint—the model receives one question and outputs a response, then stops; but an agent working is more like a relay race.

It breaks down a goal into dozens or even hundreds of steps—reading files, writing code, running commands, reviewing results, and then deciding the next step, one after another, until the task is truly completed.

Over the course of this process, dozens or even hundreds of large model calls are chained together, each passing along an increasingly lengthy context to the next step, alongside tool calls for compilation, database queries, and search execution.

Complexity is not simply additive; it is multiplicative at each level.

NVIDIA uses the metaphor of a "relay race" to describe agent workloads. A goal is broken down into dozens or hundreds of steps, with large model calls and tool calls passed like batons in sequence, forming an ever-lengthening chain.

The problem lies precisely here.

Existing reasoning benchmarks on the market measure single invocations: how long it takes for a response to return after a single request, and how many requests a single machine can handle simultaneously.

They were never designed for agents. Chain calls, tool waiting, context inflation—these place entirely different demands on the system compared to single requests.

Even in long conversations, the old benchmark's blind spot lies hidden: the same long prefix repeats round after round—those who can cache it and avoid recalculating it each time save significant computational power.

Additionally, when tools produce results that frequently overwhelm the context while the output often consists of only a few hundred tokens, whether the scheduler and memory hierarchy can handle this erratic, fluctuating pattern directly determines whether the entire system operates smoothly or crashes outright.

This is precisely where fixed-length synthetic tests fall short.

For those who buy cards and build data centers with real money, what truly matters is how many working agents this system can support simultaneously, and how much useful output each kilowatt-hour of electricity and each GPU generates.

These questions stumped the old benchmark tests.

The first ruler made for agents

AA-AgentPerf's approach differs from the old benchmark: instead of feeding fixed-length synthetic prompts, it replays actual programming agent trajectories.

Illustration of agent trajectories in AA-AgentPerf replay. Starting from a request, LLM calls and tool calls alternate until the task is fully completed.

These trajectories were collected by having agents solve problems from real code repositories, covering over 12 programming languages, with individual conversations reaching up to 200 turns and context lengths easily exceeding 100,000 tokens.

Input lengths range from 5,000 to 130,000 tokens, averaging around 27,000. The length is primarily driven not by the prompts themselves, but by the accumulated tool outputs and conversation history across multiple rounds.

More importantly, how does it calculate performance?

It doesn’t chase extreme concurrency. Once concurrency gets too high, each agent slows down to a crawl—no matter how large the number, it becomes impressive in theory but useless in practice.

Reverse AA-AgentPerf: First, establish a fixed service standard—every agent must meet specified benchmarks for output speed and time to first token (TTFT)—then determine how many agents the system can support while maintaining this threshold.

This set of constraints is called a Service Level Objective (SLO).

This standard is divided into several tiers, ranging from a baseline of 20 tokens per second to a high-speed tier of 180 tokens per second, with maximum concurrency tested separately for each tier to reflect real-world service levels available on the market.

How the Service Level Objective (SLO) caps maximum concurrency: Green dots indicate the target zone; once concurrency rises and speed drops below the threshold, the corresponding concurrency limit represents the system's performance.

It also does something that other benchmarks rarely dare to do: it fully enables all the optimizations that manufacturers actually use in production.

KV cache reuse, speculative decoding, and separating prefilling and decoding for independent deployment—techniques previously often disabled by benchmarks—are all allowed this time.

The reason is simple: turning off these optimizations renders the measurements meaningless.

At the same time, it closely monitors output quality, ensuring that no optimization gains higher concurrency at the expense of answer quality. This way, every improvement brought by advancements in software or hardware is accurately measured.

Ultimately, it comes down to one core metric: agents per megawatt. In a world where electricity is becoming increasingly scarce and energy consumption equals cost, this is the metric that buyers truly care about: shifting from tokens per second to agents per megawatt.

20 times ahead per megawatt

Each GPU is 40 times faster.

In a benchmark testing the most advanced class of modern mixture-of-experts (MoE) models, the GB300 NVL72 supports 61,400 concurrent agents per megawatt, averaging 57.5 agents per GPU.

The control group H200 has approximately 2,600 per megawatt and only 1.4 per GPU. There is about a 20-fold difference per megawatt and a 40-fold difference per GPU between the two.

These two numbers also have different levels of value.

Per megawatt measures how much agent capacity you can buy with the same amount of electricity—a measure of energy efficiency; per GPU measures the service density of a single card—a measure of hardware performance.

With these two numbers, you can directly calculate how large an agent application your available electricity budget can support.

The leaderboard features not only NVIDIA’s GB300 but also AMD’s MI355X, with single cards, complete systems, and entire racks all displayed side by side in competition.

The first batch of results revealed two clear patterns.

Rule 1: Rack-level systems are inherently more cost-effective, as they can better distribute inference workloads across more GPUs, outperforming single-node systems in both raw computational power and efficiency per megawatt.

Rule 2: The leap from Hopper to Blackwell has elevated the system’s concurrent capacity to an entirely new level—not just minor improvements.

From single-card to rack-level system victory

From the H200 to the GB300, this appears to be a leap in single-GPU performance—it’s actually a system-level victory.

More importantly, the GB300 NVL72 connects 72 GPUs into a single rack-scale unit using NVLink.

For such a large mixture-of-experts model, the key is this: the model can be fully distributed, with experts allocated across entire GPUs to execute in parallel, rather than being crammed onto a single card.

The CUDA cores have been further optimized to overlap communication and computation across experts, quietly absorbing the overhead of coordinating experts into computational power rather than adding to latency.

TensorRT-LLM ensures efficiency as concurrent sessions continue to increase, for example, by separating input processing from output generation and optimizing each independently.

In simple terms, this test result is the combined outcome of hardware, connectivity, and the software stack.

GB300 NVL72 rack. Seventy-two GPUs connected via NVLink to form a single high-bandwidth unit—this is the hardware foundation that enables 60,000 agents to operate in coordination.

Soldering 72 cards together into a single high-bandwidth unit enables each GPU to rapidly share parameters, KV caches, and intermediate results—this is the foundation that allows 60,000 agents to run in coordination.

Several boundaries that cannot be overlooked

There are a few points to note: do not equate benchmarking with real-world production conditions.

First, the number 60,000 does not mean that a single machine is running 60,000 independent large models simultaneously.

This is a concurrent session simulation under the baseline definition, where each agent follows a pre-recorded trajectory, and tool calls are not actually executed but instead simulated with a fixed amount of CPU time.

This design is intended to ensure that the final result reflects only differences in hashing power, but it cannot be directly equated to the actual service capacity deliverable in a real production environment.

Second, the benchmark performance is not a production service agreement.

Artificial Analysis itself acknowledges that this is a rapidly evolving snapshot of cutting-edge performance, with each system still having untapped potential; results will continue to improve as software optimizations advance.

Third, AA-AgentPerf is currently a standard proposed by a single institution.

It’s too early to tell whether it will eventually become an industry-wide standard like MLPerf.