MiMo-V2-Flash

Technical Overview

MiMo‑V2‑Flash is a Mixture‑of‑Experts (MoE) language model released by the XiaomiMiMo team. It packs a massive 309 billion total parameters while only 15 billion of them are active at inference time, making it both powerful and cost‑effective. The model is built for high‑speed reasoning, long‑context processing (up to 256 k tokens) and agentic workflows such as tool‑use, planning and reinforcement‑learning‑based rollout.

Key features and capabilities

• Hybrid Attention Architecture – interleaves Sliding‑Window Attention (SWA) with Global Attention (GA) in a 5:1 ratio using a very aggressive 128‑token window. This reduces KV‑cache memory by ~6× while preserving long‑range dependencies via a learnable attention sink bias.
• Multi‑Token Prediction (MTP) – a lightweight 0.33 B‑parameter per‑block dense FFN that predicts several tokens in parallel, tripling inference throughput without sacrificing quality.
• FP8 Mixed‑Precision Pre‑Training – 27 trillion tokens were consumed with FP8, cutting training cost dramatically and enabling a native 32 k sequence length.
• Agentic Distillation – post‑training Multi‑Teacher On‑Policy Distillation (MOPD) and large‑scale agentic RL give the model a measurable edge on benchmarks that test tool use and complex reasoning (e.g., SWE‑Bench).

Architecture highlights

• Mixture‑of‑Experts routing with a 15 B active sub‑network; each token is processed by a subset of expert FFNs.
• Hybrid attention layer that alternates 5 blocks of SWA with 1 block of GA, dramatically shrinking the KV‑cache footprint.
• Multi‑Token Prediction head that emits up to three tokens per forward pass, boosting real‑time responsiveness.
• Support for up to 256 k context tokens via a hierarchical attention‑sink mechanism.

Intended use cases

• Interactive chat agents that need fast turn‑around and long memory.
• Tool‑augmented assistants (code generation, web‑search, data‑analysis) where low latency is critical.
• Reinforcement‑learning pipelines that benefit from rapid rollout, thanks to MTP.
• Research on MoE scaling, hybrid attention, and agentic RL.

Benchmark Performance

MiMo‑V2‑Flash is evaluated on a suite of standard language‑model benchmarks that stress reasoning, knowledge retrieval and multi‑step problem solving. The most relevant tests for a MoE, long‑context model are BBH, MMLU (including Redux and Pro variants), DROP, ARC‑Challenge and HellaSwag. These benchmarks cover a mix of multiple‑choice, open‑ended and arithmetic tasks, providing a holistic view of both factual accuracy and chain‑of‑thought reasoning.

The results (Base variant) show:

• BBH 3‑shot: 88.5 % – on par with larger 32 B‑parameter MoE models.
• MMLU 5‑shot: 86.7 % – slightly below the 87.8 % of a 32 B‑parameter Kimi‑K2, yet superior to many dense 15 B baselines.
• MMLU‑Redux 5‑shot: 90.6 % – best among the listed competitors.
• DROP 3‑shot: 84.7 % – competitive with 37 B‑parameter dense models.
• ARC‑Challenge 25‑shot: 95.9 % – within 0.3 % of the top‑performing 32 B‑parameter model.
• HellaSwag 10‑shot: 88.5 % – modest but acceptable given the model’s inference efficiency.

These scores demonstrate that despite activating only 15 B parameters, MiMo‑V2‑Flash matches or exceeds the performance of denser models on a broad spectrum of tasks, validating the effectiveness of its hybrid attention and MTP design.

Hardware Requirements

Running the active 15 B sub‑network of MiMo‑V2‑Flash still demands high‑end hardware. For a single‑GPU inference setup, a 48 GB VRAM accelerator (e.g., NVIDIA RTX 4090, A6000) is the minimum to hold the active expert weights, KV‑cache for a 32 k context, and the MTP buffers. For the full 309 B MoE routing table, a multi‑GPU configuration (≥ 8 × 40 GB) is required to store all expert parameters, though only a fraction is accessed per token.

Recommended GPU specifications

• GPU: NVIDIA H100 (80 GB) or AMD MI250X – ideal for FP8 inference and large KV‑cache.
• PCIe 4.0 or NVLink interconnect for multi‑GPU expert sharding.
• CPU: 16‑core Xeon/AMD EPYC with ≥ 64 GB RAM for token preprocessing and routing logic.
• Storage: 200 GB SSD (NVMe) to hold the model checkpoint, MTP weights and auxiliary files.

Performance: the MTP module can generate up to three tokens per forward pass, delivering a ~3× speed‑up compared with dense 15 B models on the same hardware. The hybrid attention reduces KV‑cache memory by ~6×, enabling longer context windows (up to 256 k tokens) without exhausting VRAM.

Use Cases

MiMo‑V2‑Flash shines in scenarios where low latency, long context, and sophisticated reasoning intersect.

• Enterprise chat assistants – handle multi‑turn conversations with up to 256 k tokens of history, enabling personalized support without losing earlier context.
• Tool‑augmented agents – integrate with code interpreters, web‑search APIs, or database query engines; the MTP head accelerates the iterative “think‑act‑observe” loop.
• Reinforcement‑learning environments – fast rollout speeds make it ideal for training autonomous agents that need to evaluate many policies per second.
• Research on MoE scaling – the hybrid attention and attention‑sink bias provide a testbed for next‑generation efficient transformers.
• Content generation – long‑form writing, report drafting, or script writing where the model can keep track of hundreds of thousands of tokens.

Training Details

MiMo‑V2‑Flash was trained on 27 trillion tokens using FP8 mixed‑precision on a cluster of NVIDIA H100 GPUs. The training pipeline employed a native 32 k sequence length and a 256 k token context window for fine‑tuning, leveraging the hybrid attention to keep memory usage tractable.

Key aspects:

• Dataset: a blend of publicly available web crawls, code repositories, and instruction‑following data (similar to the mix used for LLaMA‑2 and DeepSeek).
• Compute: estimated at several thousand GPU‑years of H100 FP8 compute (≈ 10 k GPU‑hours).
• Fine‑tuning: the model ships with a 3‑layer MTP checkpoint that can be fine‑tuned on domain‑specific data using standard Hugging Face Trainer APIs.
• RL & Distillation: post‑training MOPD and agentic RL were applied on top of the base checkpoint, improving performance on tool‑use benchmarks such as SWE‑Bench.

Related Papers

The project links to a technical report PDF (the primary research paper) and a blog post on the Xiaomi MiMo site. The paper describes the hybrid attention mechanism, the MTP module, and the Multi‑Teacher On‑Policy Distillation (MOPD) pipeline. It also references foundational works such as:

• “Switch Transformers” – the seminal MoE architecture that inspired the routing strategy.
• “Long‑Range Arena (LRA)” – motivating the need for efficient long‑context attention.
• “Mistral 7B” – for dense FFN design used in the MTP block.

These references collectively underpin the design choices that make MiMo‑V2‑Flash both scalable and fast.

Licensing Information

The repository’s README states a MIT license, but the overall model card lists the license as unknown. In practice, the MIT statement grants you broad freedoms: you may use, modify, distribute and even commercialize the code and model weights, provided you retain the original copyright notice and license text.

If you decide to deploy the model commercially, you should:

• Include the MIT copyright header in any redistributed binaries or source.
• Verify that any third‑party data used for fine‑tuning respects its own licenses.
• Check the “unknown” tag on the model card for any additional restrictions that the model provider might impose (e.g., usage‑policy or export controls).

In short, the MIT clause is permissive, but you must still respect any platform‑level policies (Hugging Face Terms of Service) and ensure that downstream applications comply with local regulations.