This could be the moment AMD has been waiting for.
Recently, Wafer AI deployed on the AMD MI355X Kimi K3. As a result, a model that previously required 16 NVIDIA B200 GPUs across two servers can now be deployed on a single AMD server equipped with 8 MI355X GPUs.
More importantly, it’s not just about putting the model inside.
In a test with an input of 1,024 tokens and an output of 400 tokens, the MI355X achieved a total throughput of 952 tokens/s and a single-user generation speed of 118 tokens/s.
Calculated on a single-node basis, its throughput is approximately 3.8 times that of the 16-card B200 solution, and its cost-performance ratio exceeds both the B200 and B300.
And most surprisingly, ROCm didn’t cause any particular hassle this time.
The model is too large; video memory has become more important than computational power.
Kimi K3 has 2.8 trillion parameters, requiring over 1.5 TB of GPU memory just for model weights, not including the KV cache needed for millions of context tokens.
An 8-GPU B200 server, with each GPU having 192 GB of VRAM, provides a total capacity of approximately 1.5 TB. This means it is difficult to fit even the model weights alone, let alone allocate space for KV Cache. Therefore, the B200 requires two servers and 16 GPUs.
Each B300 card has 288 GB of VRAM, allowing the model to fit within a single node. Interestingly, the AMD MI355X also has 288 GB of VRAM; eight MI355X cards total approximately 2.3 TB, which can be accommodated in a single server.
This isn't just about using one fewer machine. After the model runs across nodes, generating each token may require network synchronization of data. Even with a RoCE v2 network operating at approximately 195 GB/s, cross-node communication still slows down decoding.
The MI355X keeps the entire model on a single node thanks to its larger memory.
In the final results, the peak total throughput of eight MI355X units reached 952 tokens/s, with a single-path generation speed of 118 tokens/s.
For comparison, the total throughput of a dual-node deployment with 16 B200s is 498 tokens/s, which equates to approximately 249 tokens/s per node.
In other words, the single-node throughput of the MI355X is approximately 3.8 times that of the average single-node throughput of a B200 dual-node deployment. In terms of single-user generation speed, the MI355X at 118 tokens/s also exceeds the B200’s 90 tokens/s.
The B300 remains the highest-performance solution available. A node with eight B300s achieves a total throughput of 1,568 tokens/s, with a single-path generation speed of 172 tokens/s, resulting in an overall throughput approximately 1.65 times that of the MI355X.
But the price change altered the conclusion. Wafer is calculated at $2.50 per card per hour for MI355X, $4.25 for B200, and $6 for B300.
Under this price assumption, MI355X offers approximately 48 tokens per dollar in peak throughput; B200 offers about 7 tokens per dollar; B300 offers about 33 tokens per dollar.
The B300 is faster, but the MI355X offers higher cost efficiency per unit. For data centers running large-scale open models, this may be more important than simply competing for peak performance.
More surprisingly, ROCm can be used almost directly.
For a long time, the biggest issue with AMD’s data center GPUs has often not been the hardware, but the software.
The same model can run directly on CUDA, but on ROCm, it may require changing the framework, adding operators, or even rewriting the underlying kernels.
But Kimi The situation with K3 is different.
AMD provided near-launch synchronization support. Wafer stated that the model can run directly on the MI355X, with subsequent efforts focused on resolving a few compatibility issues and optimizing performance.
One issue arises during the speculative decoding phase. Kimi K3 does not provide the draft model parameters required by MTP or EAGLE, so Wafer uses an external block diffusion draft model.
This solution runs directly on CUDA, but in the ROCm environment, the first real request causes the scheduler to throw an error due to a missing function named top_k_renorm_prob in the ROCm branch.
The function does something straightforward: selects the top k values from a probability distribution, sets all other probabilities to zero, and then renormalizes the remaining probabilities.
Wafer ultimately filled in this logic using a standard PyTorch function, eliminating the need to write custom GPU kernels or redesign the speculative decoding system.
After the fix, speculative decoding improved single-stream performance by approximately 2.2x, single-stream performance under moderate concurrency by approximately 1.7x, and peak total throughput by approximately 18%.
More importantly, the system can achieve peak throughput under higher concurrency.
The first digit was too slow, so only four zeros were added at the end.
Of course, throughput isn't the whole story for inference services. For real users, another metric that directly impacts experience is TTFT—the time waited from sending a request to seeing the first token.
On this metric, the MI355X initially performed poorly. Facing a cold-start prefill task of approximately 172,000 tokens, the MI355X required about 51 seconds, while the B300 needed only about 23 seconds.
In models that support millions of tokens in context, the prefilling task can be extremely large. If users must wait tens of seconds or longer each time when processing long contexts, even the highest decoding speeds struggle to compensate for the poor user experience.
Wafer eventually found that the performance gap stemmed almost entirely from a single attention kernel. Kimi Under an 8-way tensor parallel configuration, each GPU receives 12 attention heads. However, the faster MLA prefill kernel in AMD AITER only supports shapes that are multiples of 4, 8, or 16.
12 heads could not be matched, so the system fell back to the slower generic Triton implementation.
The solution is straightforward: pad the 12 attention heads with zeros to reach 16, invoke the existing high-performance kernel, and then retrieve only the original 12 heads after computation. No changes were made to the model architecture, nor were any new assembly kernels written—only four zeros were added.
After optimization, the stable prefill speed of the AITER MLA kernel reached approximately 13,000 tokens/s, while the original Triton fallback path was only around 4,000 to 7,000 tokens/s, reducing cold prefill time by approximately two to three times.
This optimization does not change the final decoding throughput but significantly reduces the time users wait for the first character to appear.
This also indicates that the seemingly large software gap between AMD and NVIDIA is sometimes not due to insufficient underlying capabilities, but rather because existing high-speed kernels have not yet been adapted to support certain new model architectures.
CUDA's moat still exists, but gaps have already emerged.
A single test certainly does not prove that AMD has fully caught up to NVIDIA.
The B200 is forced to run across nodes due to insufficient video memory; the B300 still leads in absolute performance; ROCm’s toolchain, framework support, and developer ecosystem remain inferior to CUDA.
However, open models are rapidly entering the trillion-parameter era. When models become so large that they cannot fit on a single server, GPU memory capacity is no longer just a number on a spec sheet—it directly impacts communication costs, deployment complexity, and final throughput.
AMD's strategy of allocating more HBM to single GPUs is becoming a practical system advantage.
If AMD can continue improving ROCm’s stability, expand support for high-performance core configurations, and provide timely day-one compatibility with new models, data centers must seriously consider these GPUs—offering lower prices, larger memory, sufficient performance, and software that no longer requires months of troubleshooting.
What do you think about this?
Reference link:
https://x.com/wafer_ai/status/2083628389903315406
https://x.com/ChiragAsarpota/status/2083864019870634151
This article is from the WeChat public account "Machine Heart" (ID: almosthuman2014), authored by someone focused on LLMs.