Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH. The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First, we harness the significant potential of publicly available web data through a meticulously engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO.
核心贡献 · Key contributions
构建了 DeepSeekMath 语料库，通过迭代 fastText 过滤从 Common Crawl 中提取 120B 数学词元，性能优于现有语料库。 Constructs DeepSeekMath Corpus, a 120B-token math dataset from Common Crawl via iterative fastText filtering, outperforming existing corpora.
提出了组相对策略优化（GRPO），一种无需评论家模型的 PPO 变体，减少内存占用并提升数学推理能力。 Introduces Group Relative Policy Optimization (GRPO), a PPO variant that eliminates the critic model, reducing memory while improving math reasoning.
DeepSeekMath-Base 7B 在 MATH 上达到 36.2%，在 GSM8K 上达到 64.2%，超越 Minerva 540B 和所有开源基础模型。 DeepSeekMath-Base 7B achieves 36.2% on MATH and 64.2% on GSM8K, surpassing Minerva 540B and all open-source base models.
DeepSeekMath-Instruct 7B 结合 GRPO 在 MATH 上达到 51.7%（无工具），接近 GPT-4 和 Gemini-Ultra 性能。 DeepSeekMath-Instruct 7B with GRPO achieves 51.7% on MATH without tools, approaching GPT-4 and Gemini-Ultra performance.
提供了 RL 方法（SFT、RFT、DPO、PPO、GRPO）的统一范式，分析了数据源、奖励函数和算法等关键组件。 Provides a unified paradigm for RL methods (SFT, RFT, DPO, PPO, GRPO), analyzing key components: data source, reward function, and algorithm.
证明了代码预训练提升数学推理能力，而 arXiv 论文在此背景下未显示显著益处。 Demonstrates code pre-training improves math reasoning, while arXiv papers show no notable benefit in this context.
局限 · Limitations
几何和定理证明能力弱于 GPT-4 等闭源模型，表明存在数据选择偏差。 Geometry and theorem-proving capabilities are weaker than closed models like GPT-4, indicating data selection bias.
DeepSeekMath 7B 在零样本和少样本下表现相似，而 GPT-4 能从少样本输入中获益。 DeepSeekMath 7B shows similar zero-shot and few-shot performance, unlike GPT-4 which benefits from few-shot inputs.
RL 改进仅限于 Maj@K 准确率，而非 Pass@K，表明其稳定输出分布而非增强基础能力。 RL improvement is limited to Maj@K accuracy, not Pass@K, suggesting it stabilizes output distribution rather than enhancing fundamental capabilities.
研究未探索 arXiv 词元对定理非形式化等特定任务或更大模型规模的影响。 The study does not explore arXiv tokens' impact on specific tasks like theorem informalization or at larger model scales.
RL 流程仅使用指令调优中的分布内问题，限制了向分布外提示的泛化能力。 RL pipeline uses only in-distribution questions from instruction tuning, limiting generalization to out-of-distribution prompts.
论文章节 · Sections（共 15）
摘要Abstract
1 引言1 Introduction
1.1 贡献1.1 Contributions
1.2 评估与指标总结1.2 Summary of Evaluations and Metrics
2.1 数据收集与去污染2.1 Data Collection and Decontamination
2.2 验证 DeepSeekMath 语料库质量2.2 Validating the Quality of the DeepSeekMath Corpus
2.3 训练与评估 DeepSeekMath-Base 7B2.3 Training and Evaluating DeepSeekMath-Base 7B
3.1 SFT 数据整理3.1 SFT Data Curation
3.2 训练与评估 DeepSeekMath-Instruct 7B3.2 Training and Evaluating DeepSeekMath-Instruct 7B
4.1 组相对策略优化4.1 Group Relative Policy Optimization
4.2 训练与评估 DeepSeekMath-RL4.2 Training and Evaluating DeepSeekMath-RL
5 讨论5 Discussion
5.1 预训练经验教训5.1 Lessons Learnt in Pre-Training
5.2 强化学习见解5.2 Insights of Reinforcement Learning
6 结论、局限与未来工作6 Conclusion, Limitation, and Future Work