We introduce Magistral, Mistral's first reasoning model and our own scalable reinforcement learning (RL) pipeline. Instead of relying on existing implementations and RL traces distilled from prior models, we follow a ground up approach, relying solely on our own models and infrastructure. Notably, we demonstrate a stack that enabled us to explore the limits of pure RL training of LLMs, present a simple method to force the reasoning language of the model, and show that RL on text data alone maintains most of the initial checkpoint's capabilities. We find that RL on text maintains or improves multimodal understanding, instruction following and function calling. We present Magistral Medium, trained for reasoning on top of Mistral Medium 3 with RL alone, and we open-source Magistral Small (Apache 2.0) which further includes cold-start data from Magistral Medium.
核心贡献 · Key contributions
提出了 Magistral，Mistral 首个通过可扩展强化学习管道从头训练的推理模型。 Introduced Magistral, Mistral's first reasoning model trained via a scalable RL pipeline from scratch.
证明了纯文本强化学习能保持或提升多模态理解、指令遵循和函数调用能力。 Demonstrated that pure RL on text data preserves or improves multimodal understanding, instruction following, and function calling.
提出了一种通过奖励语言一致性来强制多语言推理的简单方法。 Presented a simple method to enforce multilingual reasoning by rewarding language consistency.
开发了异步分布式强化学习基础设施，支持频繁权重更新的高效在策略训练。 Developed an asynchronous distributed RL infrastructure enabling efficient on-policy training with frequent weight updates.
展示了仅用强化学习在 24B 模型上能达到与基于蒸馏方法相当的性能。 Showed that RL alone on a 24B model achieves performance comparable to distillation-based methods.
以 Apache 2.0 许可证开源了 Magistral Small（24B）模型。 Open-sourced Magistral Small (24B) under Apache 2.0 license.
局限 · Limitations
强化学习训练依赖可验证奖励，限制了其在需要客观正确性任务之外的适用性。 RL training relies on verifiable rewards, limiting applicability to tasks with objective correctness.
由于语言约束，多语言性能相比英语下降 4-10%。 Multilingual performance degrades 4-10% compared to English due to language constraints.
该方法需要大量数据过滤和两阶段难度筛选。 The method requires extensive data filtering and two-stage difficulty curation.
代码任务的比例奖励导致最终性能略低于二元奖励。 Proportional rewards for code tasks led to slightly lower final performance than binary rewards.
熵奖励不稳定，被 Clip-Higher 策略取代以控制探索。 Entropy bonus was unstable and replaced by Clip-Higher strategy for exploration control.
论文章节 · Sections（共 28）
摘要Abstract
1 引言1 Introduction
2 方法2 Methodology
2.1 强化学习算法2.1 Reinforcement learning algorithm
2.2 奖励塑形2.2 Reward shaping
3 基础设施3 Infrastructure
4 数据整理4 Data curation
4.1 数学4.1 Math
4.2 代码4.2 Code
5 实验与结果5 Experiment and results
5.1 评估基准与基线5.1 Evaluation benchmarks and baselines
5.2 Magistral Medium——从零开始的推理 RL5.2 Magistral Medium – reasoning RL from scratch
5.3 Magistral Small——基于推理 SFT 引导的 RL5.3 Magistral Small – RL on top of reasoning SFT bootstrapping
5.4 多语言基准5.4 Multilingual benchmarks
6 消融实验6 Ablations
6.1 跨领域泛化6.1 Cross-domain generalization
6.2 小模型的蒸馏与 RL 对比6.2 Distillation vs. RL for small models
6.3 批次与小批量大小6.3 Batch and minibatch size
6.4 优势归一化6.4 Advantage normalization
7 分析7 Analysis
7.1 强化学习在低维空间中移动权重7.1 Reinforcement learning moves weights in low-dimensional space
7.2 享用多模态免费午餐7.2 Eating the multimodal free lunch
7.3 RL 对其他能力的影响7.3 Impact of RL on other capabilities
7.4 不成功的方法7.4 Unsuccessful approaches
8 在基于开源推理轨迹微调的模型上应用 RL8 RL on model finetuned using OSS reasoning traces