Xiaomi's MiMo-V3 Architecture Reduces Prefill Compute by 80%

ME AI News, Dongcha Beating AI Bulletin: Xiaomi's MiMo team has revealed that the next-generation MiMo-V3 will adopt the HySparse2 architecture, primarily addressing the issue of increasing costs as Agents run longer. Agents often issue very short commands but may retrieve large volumes of content from web pages, terminals, and tools. The model must first process all this new input; the longer the context, the more computational power this step consumes and the more VRAM is occupied by KV Cache. HySparse2 splits the model into two segments: the first handles input processing, while the second continues inference and generation. The context needed by the second segment can be directly generated from results already computed by the first segment, eliminating the need to fully re-run the second segment during prefill. This approach originates from Microsoft Research’s 2024 YOCO (You Only Cache Once) method, which enables the second segment to reuse previously computed information, reducing both cache storage and redundant computation. HySparse2 also redesigns sparse attention: while standard full attention examines the entire context each time, HySparse2 limits this to only a few layers. These layers first select the 1,024 most relevant tokens from the long context; subsequent layers reuse this selection while permanently retaining the most recent 128 tokens. In the previous HySparse version, tokens were grouped in sets of 64, and if any token in a group was important, the entire group was preserved. HySparse2 now selects tokens individually, allowing the same computational budget to be allocated to more truly relevant content. When tested independently, this change improved long-text retrieval scores by 6.57 and 8.14 points respectively. The paper uses a model with 80 billion total parameters and approximately 3 billion activated parameters per inference as a baseline. In a 49-layer model, only the first 25 layers need to be executed during prefill. When input reaches 1 million tokens, prefill computation is reduced to about 1/5 compared to MiMo-V2’s hybrid sliding window attention, and KV Cache drops from 12.09 GB to 2.69 GB. (Source: BlockBeats)