Qwen2.5-Math 技术报告：通过自我改进迈向数学专家模型

Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement

杨安 An Yang · Qwen Team, Alibaba · 2024-09-18 · arXiv:2409.12122 ↗ · 被引 1029

摘要 · Abstract

本报告介绍了一系列数学专用大语言模型：Qwen2.5-Math 和 Qwen2.5-Math-Instruct-1.5B/7B/72B。Qwen2.5 系列的核心创新在于将自我改进理念贯穿整个流程，从预训练、后训练到推理：（1）在预训练阶段，利用 Qwen2-Math-Instruct 生成大规模、高质量的数学数据。（2）在后训练阶段，我们通过对 Qwen2-Math-Instruct 进行大量采样来开发奖励模型（RM）。然后将该 RM 应用于监督微调（SFT）数据的迭代演化。有了更强的 SFT 模型，就可以迭代训练和更新 RM，进而指导下一轮 SFT 数据迭代。在最终的 SFT 模型上，我们使用最终的 RM 进行强化学习，得到 Qwen2.5-Math-Instruct。（3）此外，在推理阶段，使用 RM 指导采样，优化模型性能。Qwen2.5-Math-Instruct 支持中文和英文，并具备先进的数学推理能力，包括思维链（CoT）和工具集成推理（TIR）。我们在 10 个英文和中文数学数据集上评估了模型，包括 GSM8K、MATH、高考、AMC23 和 AIME24，涵盖了从小学水平到数学竞赛问题的各种难度。

In this report, we present a series of math-specific large language models: Qwen2.5-Math and Qwen2.5-Math-Instruct-1.5B/7B/72B. The core innovation of the Qwen2.5 series lies in integrating the philosophy of self-improvement throughout the entire pipeline, from pre-training and post-training to inference: (1) During the pre-training phase, Qwen2-Math-Instruct is utilized to generate large-scale, high-quality mathematical data. (2) In the post-training phase, we develop a reward model (RM) by conducting massive sampling from Qwen2-Math-Instruct. This RM is then applied to the iterative evolution of data in supervised fine-tuning (SFT). With a stronger SFT model, it's possible to iteratively train and update the RM, which in turn guides the next round of SFT data iteration. On the final SFT model, we employ the ultimate RM for reinforcement learning, resulting in the Qwen2.5-Math-Instruct. (3) Furthermore, during the inference stage, the RM is used to guide sampling, optimizing the model's performance. Qwen2.5-Math-Instruct supports both Chinese and English, and possess advanced mathematical reasoning capabilities, including Chain-of-Thought (CoT) and Tool-Integrated Reasoning (TIR). We evaluate our models on 10 mathematics datasets in both English and Chinese, such as GSM8K, MATH, GaoKao, AMC23, and AIME24, covering a range of difficulties from grade school level to math competition problems.

核心贡献 · Key contributions

在预训练、后训练和推理阶段整合自我改进，利用合成数据和奖励模型。
Integrates self-improvement across pre-training, post-training, and inference using synthetic data and reward models.
推出支持中英文思维链和工具集成推理的 Qwen2.5-Math-Instruct。
Introduces Qwen2.5-Math-Instruct supporting Chain-of-Thought and Tool-Integrated Reasoning in English and Chinese.
开发迭代奖励模型训练和数据演化，用于监督微调和强化学习。
Develops iterative reward model training and data evolution for supervised fine-tuning and reinforcement learning.
在 MATH、AMC 2023 和 AIME 2024 上取得最先进结果，超越 GPT-4o 和 Gemini。
Achieves state-of-the-art results on MATH, AMC 2023, and AIME 2024, outperforming GPT-4o and Gemini.
展示跨尺度改进：1.5B 模型超越多数 70B 模型，7B 匹配先前 72B 性能。
Demonstrates cross-scale improvement: 1.5B model surpasses most 70B models, 7B matches previous 72B.
在 Hugging Face 和 GitHub 上提供开源模型、奖励模型和评估脚本。
Provides open-source models, reward models, and evaluation scripts on Hugging Face and GitHub.

局限 · Limitations

评估限于 10 个数学数据集，未验证对其他领域的泛化能力。
Evaluation limited to 10 math datasets; generalization to other domains not verified.
工具集成推理在中文基准上相比思维链无显著优势。
Tool-Integrated Reasoning shows no significant advantage over CoT in Chinese benchmarks.
依赖 Qwen2-Math 的合成数据可能引入偏差或质量问题。
Dependence on synthetic data from Qwen2-Math may introduce biases or quality issues.
强化学习需要大规模采样和奖励模型，增加计算成本。
Reinforcement learning requires large-scale sampling and reward model, increasing computational cost.
去污染可能无法完全消除训练集中相似问题的数据泄露。
Decontamination may not fully eliminate data leakage from similar problems in training sets.

论文章节 · Sections（共 12）

摘要 Abstract
1 引言 1 Introduction
2 Qwen2.5-Math 预训练 2 Qwen2.5-Math Pre-training
3 Qwen2.5-Math 后训练 3 Qwen2.5-Math Post-training
3.1 监督微调 3.1 Supervised Fine-tuning
3.2 奖励模型训练 3.2 Reward Model Training
3.3 强化学习 3.3 Reinforcement Learning
4 数据去污染 4 Decontamination
5.1 基础模型 5.1 Base Models
5.2 指令模型 5.2 Instruction Models
6 结论 6 Conclusion
致谢 Acknowledgements

阅读逐段中英对照全文 →