Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale. We introduce MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA). A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group, enabling group-specific sparse retrieval while maintaining efficient block-level execution; the Main Branch then performs exact block-sparse attention over only the selected blocks. Designed around a principle of simplicity and scalability, MSA is deliberately streamlined, making it straightforward to deploy efficiently across a broad range of GPUs. To translate sparsity into practical speedups, we co-design MSA with a GPU execution path that uses exp-free Top-k selection and KV-outer sparse attention to improve tensor-core utilization under block-granular access. On a 109B-parameter model with native multimodal training, MSA performs on par with GQA while reducing per-token attention compute by 28.4x at 1M context. Paired with our co-designed kernel, MSA achieves 14.2x prefill and 7.6x decoding wall-clock speedups on H800. Our inference kernel is available at: https://github.com/MiniMax-AI/MSA. A production-grade natively multimodal model powered by MSA has been publicly released at: https://huggingface.co/MiniMaxAI/MiniMax-M3.
核心贡献 · Key contributions
提出 MiniMax 稀疏注意力（MSA），一种基于分组查询注意力（GQA）的块级稀疏注意力机制。 Introduces MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA).
设计轻量级索引分支，为每个 GQA 组独立选择 Top-k 键值块。 Designs a lightweight Index Branch that independently selects Top-k key-value blocks per GQA group.
协同设计 GPU 内核，采用无指数 Top-k 选择和 KV 外循环稀疏注意力实现实际加速。 Co-designs GPU kernels with exp-free Top-k selection and KV-outer sparse attention for practical speedups.
在 109B 参数模型上，1M 上下文下实现 28.4 倍注意力计算减少和 14.2 倍预填充加速。 Demonstrates 28.4x attention compute reduction and 14.2x prefill speedup on a 109B-parameter model at 1M context.
验证 MSA 在下游基准测试中与 GQA 性能相当，同时实现高效长上下文推理。 Validates MSA matches GQA on downstream benchmarks while enabling efficient long-context inference.
公开发布生产级多模态模型和推理内核。 Releases production-grade multimodal model and inference kernel publicly.
局限 · Limitations
稀疏注意力预算固定为每查询 2048 个 token，限制了极长上下文中的召回率。 Sparse attention budget fixed at 2,048 tokens per query, limiting recall for very long contexts.
需要两阶段预热训练索引器，增加了训练流程的复杂性。 Requires two-stage warmup training for indexer, adding complexity to training pipeline.
由于内核开销，运行时加速小于理论 FLOP 减少。 Runtime speedup smaller than theoretical FLOP reduction due to kernel overheads.
评估限于 109B MoE 模型，未充分探索对其他架构的泛化性。 Evaluation limited to 109B MoE model; generalization to other architectures not fully explored.
基于 KL 损失的索引器训练可能无法捕获所有注意力模式，尤其是对罕见 token。 Indexer trained with KL loss may not capture all attention patterns, especially for rare tokens.
论文章节 · Sections（共 19）
摘要Abstract
1 引言1 Introduction
2.1 因果注意力与 GQA2.1 Causal Attention and GQA
2.2 稀疏注意力作为两阶段过程2.2 Sparse Attention as a Two-Stage Process