Xiaomi's MiMo-V3 Core Component HySparse2 Slashes Prefill Compute to ~1/5 at 1M Context

On the evening of September 23, Fuli Luo, head of Xiaomi's MiMo large language model team, revealed the new architecture of MiMo-V3 ahead of its full release, introducing HySparse2 as a core component and publishing the accompanying technical paper. The architecture targets three key challenges in long-horizon agent tasks: excessive computation from long inputs, large cache footprint, and difficulty in retrieving relevant information from long contexts.

According to the paper, compared with the Hybrid SWA architecture used in MiMo-V2.6, HySparse2 reduces prefill computation to about one-fifth and KV cache usage to about one-fourth at a 1-million-token context, while improving long-context retrieval and lowering AgentPPL and LongPPL.

The efficiency gain comes from two-level KV sharing, comprising KV Bridging and KV Reuse. The model is split into a Self-Decoder and a Cross-Decoder, allowing the latter to reuse hidden states and KV caches generated by the former, so prefill can exit after running roughly half the layers. HySparse2 also replaces block-level token selection with token-level selection, enabling more direct retrieval of relevant tokens across long contexts.

In experiments with an 80B-A3B MoE model, HySparse2 achieved a RULER-v2 score of 58.45 at a 256K context, compared with 32.61 for HySparse and 35.74 for Hybrid SWA. General capabilities were roughly on par across the three architectures.

The paper was completed by Xiaomi's LLM-Core team with 15 authors, with Fuli Luo as corresponding author. The disclosure came just two days after Xiaomi open-sourced the MiMo-V2.6 series, offering an early look at the underlying architecture direction of MiMo-V3.