The success of DeepSeek-R1 underscores the significant role of reinforcement learning (RL) in enhancing the reasoning capabilities of large language models (LLMs). In this work, we present Skywork-OR1, an effective and scalable RL implementation for long Chain-of-Thought (CoT) models. Building on the DeepSeek-R1-Distill model series, our RL approach achieves notable performance gains, increasing average accuracy across AIME24, AIME25, and LiveCodeBench from 57.8% to 72.8% (+15.0%) for the 32B model and from 43.6% to 57.5% (+13.9%) for the 7B model. Our Skywork-OR1-32B model surpasses both DeepSeek-R1 and Qwen3-32B on the AIME24 and AIME25 benchmarks, while achieving comparable results on LiveCodeBench. The Skywork-OR1-7B and Skywork-OR1-Math-7B models demonstrate competitive reasoning capabilities among models of similar size. We perform comprehensive ablation studies on the core components of our training pipeline to validate their effectiveness. Additionally, we thoroughly investigate the phenomenon of entropy collapse, identify key factors affecting entropy dynamics, and demonstrate that mitigating premature entropy collapse is critical for improved test performance. To support community research, we fully open-source our model weights, training code, and training datasets.
核心贡献 · Key contributions
提出 MAGIC 框架，一种多阶段自适应熵调度的 GRPO 方法，提升长思维链推理能力。 Proposes MAGIC, a multi-stage adaptive entropy scheduling framework for GRPO, improving long CoT reasoning.
在 32B 和 7B 模型上，AIME24、AIME25 和 LiveCodeBench 平均准确率分别提升 15.0%和 13.9%。 Achieves 15.0% and 13.9% average accuracy gains on 32B and 7B models over AIME24, AIME25, and LiveCodeBench.
发现过早熵坍塌是限制强化学习性能的关键因素，并提出自适应熵控制方法。 Identifies premature entropy collapse as a key factor limiting RL performance and proposes adaptive entropy control.
对数据、训练策略和损失组件进行全面的消融研究，验证其有效性。 Conducts comprehensive ablation studies on data, training strategy, and loss components to validate effectiveness.
开源模型权重、训练代码和数据集，支持社区研究。 Open-sources model weights, training code, and datasets to support community research.
证明在线策略更新和高温度采样可缓解熵坍塌并提升泛化能力。 Demonstrates that on-policy updates and high-temperature sampling mitigate entropy collapse and improve generalization.
局限 · Limitations
实验仅限于 DeepSeek-R1-Distill 模型系列，可能不适用于其他架构。 Experiments are limited to DeepSeek-R1-Distill model series and may not generalize to other architectures.
代码沙箱无法处理非确定性输出或同一输入的多个有效答案。 Code sandbox does not handle non-deterministic outputs or multiple valid answers for the same input.
当每训练步的 SGD 步数较大时，自适应熵控制可能变得不稳定。 Adaptive entropy control may become unstable when the number of SGD steps per training step is large.
研究聚焦数学和代码领域，未评估在其他领域（如推理、自然语言处理）的有效性。 The study focuses on math and code domains; effectiveness in other domains (e.g., reasoning, NLP) is not evaluated.
训练资源分配分析假设同步训练，异步设置可能产生不同结果。 Training resource allocation analysis assumes synchronous training; asynchronous settings may yield different results.
论文章节 · Sections（共 25）
摘要Abstract
1 引言1 Introduction
2 预备知识2 Preliminaries
3 Skywork-OR1 中的 MAGIC3 MAGIC in Skywork-OR1
3.1 MAGIC3.1 MAGIC
3.2 MAGIC 组件的有效性3.2 Effectiveness of MAGIC Components
4 缓解策略熵崩溃的实证研究4 Empirical Studies on Mitigating Policy Entropy Collapse
4.1 消融实验设置4.1 Ablation Setup
4.2 过早熵崩溃通常表现为性能下降4.2 Premature Entropy Collapse Generally Manifests as Worse Performance
4.3 展开多样性相关超参数的影响4.3 The Impact of Rollout-Diversity-Related Hyperparameters
4.4 通过增加 NSGDN_{\text{SGD}}进行离策略更新的影响4.4 The Impact of Off-policy Update by Increasing NSGDN_{\text{SGD}}