SemiAnalysis Report: Memory Bandwidth Outweighs Capacity in AI Inference

ME AI News, Dongcha Beating AI Bulletin: Research firm SemiAnalysis has released a report dissecting the underlying architecture of large model inference services. The report argues that as MoE models become mainstream, AI inference has evolved from a single computational task into a complex pipeline comprising Prefill, Midfill, Decode Attention, and Decode Experts—each stage exhibiting distinct demands for compute, memory bandwidth, and network resources. In most inference scenarios, memory bandwidth is more economically valuable than capacity; high-bandwidth memory enhances token generation efficiency, while idle HBM only increases costs. SemiAnalysis estimates that by 2027, each pipeline stage may require approximately 400 to 500 GB of local fast memory, but completed KV caches should be promptly migrated to CPU DRAM and lower-cost network storage to avoid consuming scarce HBM resources. The scheduling layer will become a critical component of AI inference infrastructure. Prefill and Midfill processing times are relatively predictable, while decode durations exhibit significant variability, leading to task backlogs and “latency feedback oscillations.” Consequently, stable buffering mechanisms, cross-time-scale scheduling, and hierarchical KV cache management will directly impact the throughput efficiency of inference clusters. Regarding architecture choices, the report compares aggregated and disaggregated approaches: the former consolidates Prefill and Decode on the same device, reducing KV cache transfers across nodes but requiring hardware with both high compute density and high memory bandwidth; the latter deploys dedicated nodes for each stage, enabling hardware optimization but increasing data movement and network overhead. SemiAnalysis believes the ultimate trade-off between these architectures depends on whether a future “all-in-one” accelerator capable of delivering both high compute and high memory bandwidth will emerge. The report also uses a simulator to predict the performance of Kimi K3 on NVIDIA B200, B300, and GB200 systems. Results show that GB200 holds advantages in certain low-latency scenarios, while peak HBM residency per GPU in most cutting-edge configurations remains below 80 GB—further underscoring the importance of memory bandwidth and storage orchestration. (Source: BlockBeats)