DAPO：一个开源的大规模 LLM 强化学习系统

DAPO: An Open-Source LLM Reinforcement Learning System at Scale

于琪颖 Qiying Yu · ByteDance Seed · 2025-03-18 · arXiv:2503.14476 ↗ · 被引 2144

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

推理扩展通过强化学习赋予大语言模型前所未有的推理能力，但最先进的推理模型的关键技术细节（如 OpenAI o1 博客和 DeepSeek R1 技术报告）被隐藏，导致社区难以复现其 RL 训练结果。我们提出了解耦裁剪与动态采样策略优化（DAPO）算法，并完全开源了一个最先进的大规模 RL 系统，该系统使用 Qwen2.5-32B 基座模型在 AIME 2024 上达到 50 分。与以往隐藏训练细节的工作不同，我们介绍了使大规模 LLM RL 成功的四项关键技术。此外，我们开源了基于 verl 框架的训练代码，以及精心整理和处理的数据集。这些开源组件增强了可复现性，并支持未来大规模 LLM RL 的研究。

Inference scaling empowers LLMs with unprecedented reasoning ability, with reinforcement learning as the core technique to elicit complex reasoning. However, key technical details of state-of-the-art reasoning LLMs are concealed (such as in OpenAI o1 blog and DeepSeek R1 technical report), thus the community still struggles to reproduce their RL training results. We propose the $\textbf{D}$ecoupled Clip and $\textbf{D}$ynamic s$\textbf{A}$mpling $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{DAPO}$) algorithm, and fully open-source a state-of-the-art large-scale RL system that achieves 50 points on AIME 2024 using Qwen2.5-32B base model. Unlike previous works that withhold training details, we introduce four key techniques of our algorithm that make large-scale LLM RL a success. In addition, we open-source our training code, which is built on the verl framework, along with a carefully curated and processed dataset. These components of our open-source system enhance reproducibility and support future research in large-scale LLM RL.

核心贡献 · Key contributions

• 提出 DAPO 算法，包含四项关键技术，用于大规模 LLM 强化学习。
  Proposes DAPO algorithm with four key techniques for large-scale LLM RL.
• 使用 Qwen2.5-32B 在 AIME 2024 上达到 50 分，超越 DeepSeek-R1-Zero-Qwen-32B。
  Achieves 50 points on AIME 2024 using Qwen2.5-32B, surpassing DeepSeek-R1-Zero-Qwen-32B.
• 开源训练代码、数据集和算法细节，提升可复现性。
  Open-sources training code, dataset, and algorithm details for reproducibility.
• 引入 Clip-Higher 策略，防止熵崩溃并促进探索。
  Introduces Clip-Higher to prevent entropy collapse and promote exploration.
• 提出动态采样，过滤零梯度提示，提高训练效率。
  Proposes Dynamic Sampling to filter zero-gradient prompts and improve efficiency.
• 实现词元级策略梯度损失和过长奖励塑造，提升训练稳定性。
  Implements Token-Level Policy Gradient Loss and Overlong Reward Shaping for stability.

局限 · Limitations

• 评估仅限于数学任务，未验证对其他领域的泛化能力。
  Evaluation limited to mathematical tasks; generalization to other domains not verified.
• 大规模强化学习训练需要大量算力资源。
  Requires significant computational resources for large-scale RL training.
• 过长奖励塑造仍可能对截断响应引入偏差。
  Overlong reward shaping may still introduce bias for truncated responses.
• 动态采样增加采样实例，可能影响吞吐量。
  Dynamic Sampling increases sampling instances, potentially affecting throughput.
• 训练奖励与验证准确率相关性弱，表明存在过拟合。
  Training reward correlates weakly with validation accuracy, indicating overfitting.

论文章节 · Sections（共 19）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2.1 近端策略优化（PPO） 2.1 Proximal Policy Optimization (PPO)
• 2.2 组相对策略优化（GRPO） 2.2 Group Relative Policy Optimization (GRPO)
• 2.3 移除 KL 散度 2.3 Removing KL Divergence
• 2.4 基于规则的奖励建模 2.4 Rule-based Reward Modeling
• 3 DAPO 3 DAPO
• 3.1 提升上限：Clip-Higher 3.1 Raise the Ceiling: Clip-Higher
• 3.2 多多益善：动态采样 3.2 The More the Merrier: Dynamic Sampling
• 3.3 再平衡：Token 级策略梯度损失 3.3 Rebalancing Act: Token-Level Policy Gradient Loss
• 3.4 捉迷藏：过长奖励塑造 3.4 Hide and Seek: Overlong Reward Shaping
• 3.5 数据集转换 3.5 Dataset Transformation
• 4.1 训练细节 4.1 Training Details
• 4.2 主要结果 4.2 Main Results
• 4.3 训练动态 4.3 Training Dynamics
• 4.4 案例研究 4.4 Case Study
• 5 结论 5 Conclusion
• 贡献 Contributions
• 致谢 Acknowledgments

阅读逐段中英对照全文 →