MiniMax-01：利用闪电注意力扩展基础模型

MiniMax-01: Scaling Foundation Models with Lightning Attention

闫俊杰 Junjie Yan · · 2025-01-14 · arXiv:2501.08313 ↗

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

我们推出了 MiniMax-01 系列，包括 MiniMax-Text-01 和 MiniMax-VL-01，它们在性能上与顶级模型相当，并在处理更长上下文方面展现出卓越能力。其核心在于闪电注意力及其高效扩展。为了最大化计算能力，我们将其与专家混合（MoE）架构结合，构建了一个拥有 32 个专家、总计 4560 亿参数的模型，其中每个 token 激活 459 亿参数。我们为 MoE 和闪电注意力开发了优化的并行策略以及高效的计算-通信重叠技术。这种方法使我们能够在跨越百万级 token 的上下文中，对数千亿参数的模型进行高效训练和推理。MiniMax-Text-01 的上下文窗口在训练时可达 100 万 tokens，并在推理时以可承受的成本外推至 400 万 tokens。我们的视觉语言模型 MiniMax-VL-01 通过使用 5120 亿视觉语言 tokens 进行持续训练而构建。在标准基准和内部基准上的实验表明，我们的模型在匹配 GPT-4o 和 Claude-3.5-Sonnet 等最先进模型性能的同时，提供了长达 20 到 32 倍的上下文窗口。我们已在 https://github.com/MiniMax-AI 公开释放 MiniMax-01。

We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, which are comparable to top-tier models while offering superior capabilities in processing longer contexts. The core lies in lightning attention and its efficient scaling. To maximize computational capacity, we integrate it with Mixture of Experts (MoE), creating a model with 32 experts and 456 billion total parameters, of which 45.9 billion are activated for each token. We develop an optimized parallel strategy and highly efficient computation-communication overlap techniques for MoE and lightning attention. This approach enables us to conduct efficient training and inference on models with hundreds of billions of parameters across contexts spanning millions of tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost. Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens. Experiments on both standard and in-house benchmarks show that our models match the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering 20-32 times longer context window. We publicly release MiniMax-01 at https://github.com/MiniMax-AI.

核心贡献 · Key contributions

• 提出了 MiniMax-Text-01 和 MiniMax-VL-01，性能对标 GPT-4o 和 Claude-3.5-Sonnet 等前沿模型。
  Introduces MiniMax-Text-01 and MiniMax-VL-01, frontier models rivaling GPT-4o and Claude-3.5-Sonnet.
• 首次大规模部署了基于 lightning attention 的线性注意力，模型总参数 456B，激活参数 45.9B。
  Achieves first large-scale deployment of linear attention via lightning attention with 456B total parameters, 45.9B active.
• 设计了融合 lightning attention 与 softmax attention 的混合架构，提升了检索与长度外推能力。
  Designs hybrid architecture combining lightning and softmax attention, enhancing retrieval and length extrapolation.
• 开发了 ETP/EDP 并行组、varlen 环形注意力及 LASP+等优化并行策略，实现高效长上下文训练。
  Develops optimized parallel strategies: ETP/EDP groups, varlen ring attention, and LASP+ for efficient long-context training.
• 训练时上下文窗口达 100 万 token，推理时可外推至 400 万 token，且成本可控。
  Extends context window to 1M tokens during training and up to 4M at inference with low-cost extrapolation.
• 开源模型权重并提供 API，上下文长度比同类顶级模型长 20-32 倍。
  Releases model weights and API, offering 20-32x longer context than comparable top-tier models.

局限 · Limitations

• 纯线性注意力检索能力弱，混合设计需周期性插入 softmax 层，增加架构复杂度。
  Pure linear attention shows weak retrieval; hybrid design requires periodic softmax layers, adding complexity.
• 缩放定律外推至 93 亿参数时不可靠，需要自定义缩放公式进行预测。
  Scaling laws become unreliable when extrapolated to 9.3B parameters, requiring custom scaling formula.
• 训练需 1500-2500 块 H800 GPU，400 万 token 推理依赖 8×80G 显存和 8-bit 量化，硬件门槛高。
  Training requires 1500-2500 H800 GPUs; inference with 4M tokens demands 8x80G memory and 8-bit quantization.
• 基准评估集中于标准任务，真实世界长上下文场景下的表现可能存在差异。
  Benchmark evaluations focus on standard tasks; real-world long-context performance may vary.
• 并非所有输入都能稳定外推至 400 万 token，RULER 评测显示极端长度下性能有所下降。
  Context extrapolation to 4M tokens is not guaranteed for all inputs; RULER results show degradation at extreme lengths.

论文章节 · Sections（共 25）

• 摘要 Abstract
• 1 请将通信发送至 model@minimaxi.com。 1 Please send correspondence to model@minimaxi.com.
• 1 Introduction 1 Introduction
• 2.1 混合专家（MoE） 2.1 Mixture of Experts
• 2.2 Linear Attention 2.2 Linear Attention
• 2.4 模型规范 2.4 Model Spec
• 3 计算优化 3 Computation Optimization
• 3.1 MoE 优化 3.1 MoE Optimization
• 3.2 长上下文优化 3.2 Long Context Optimization
• 3.3 闪电注意力推理优化 3.3 Lightning Attention Inference Optimization
• 4 预训练 4 Pre-Training
• 4.1 数据 4.1 Data
• 4.2 Training Strategy 4.2 Training Strategy
• 5 后训练 5 Post-training
• 5.1 提示词收集 5.1 Prompt Collection
• 5.2 奖励模型 5.2 Reward Model
• 5.3 监督微调 5.3 Supervised Fine-Tuning
• 5.5 安全对齐 5.5 Safety Alignment
• 5.6 长上下文自适应的训练方法 5.6 Training Methodology with Long-Context Adaptation
• 5.7 Academic Benchmarks 5.7 Academic Benchmarks
• 0.4 GPT-4o（11-20） 0.4 GPT-4o (11-20)
• 5.8 用户参与 5.8 User-in-the-loop
• 6 视觉-语言模型 6 Vision-language Model
• 6.1 多模态数据 6.1 Multimodal Data
• 注 Note

阅读逐段中英对照全文 →