Recent advances in reasoning-centric language models have highlighted reinforcement learning (RL) as a promising method for aligning models with verifiable rewards. However, it remains contentious whether RL truly expands a model's reasoning capabilities or merely amplifies high-reward outputs already latent in the base model's distribution, and whether continually scaling up RL compute reliably leads to improved reasoning performance. In this work, we challenge prevailing assumptions by demonstrating that prolonged RL (ProRL) training can uncover novel reasoning strategies that are inaccessible to base models, even under extensive sampling. We introduce ProRL, a novel training methodology that incorporates KL divergence control, reference policy resetting, and a diverse suite of tasks. Our empirical analysis reveals that RL-trained models consistently outperform base models across a wide range of pass@k evaluations, including scenarios where base models fail entirely regardless of the number of attempts. We further show that reasoning boundary improvements correlates strongly with task competence of base model and training duration, suggesting that RL can explore and populate new regions of solution space over time. These findings offer new insights into the conditions under which RL meaningfully expands reasoning boundaries in language models and establish a foundation for future work on long-horizon RL for reasoning. We release model weights to support further research: https://huggingface.co/nvidia/Nemotron-Research-Reasoning-Qwen-1.5B
核心贡献 · Key contributions
提出 ProRL，一种包含 KL 散度控制和参考策略重置的长时间强化学习训练方法。 Proposes ProRL, a prolonged RL training methodology with KL divergence control and reference policy resetting.
证明长时间强化学习训练能扩展推理边界，超越基础模型能力。 Demonstrates that prolonged RL training expands reasoning boundaries beyond base model capabilities.
在 1.5B 通用推理模型上实现跨多个领域的最优性能。 Achieves state-of-the-art performance on a 1.5B generalist reasoning model across diverse domains.
表明强化学习增益在基础模型最初表现不佳的任务上最为显著。 Shows that RL gains are most significant on tasks where base model initially struggles.
提供证据表明长时间强化学习训练中涌现出新的推理策略。 Provides evidence of novel reasoning strategies emerging through prolonged RL training.
发布模型权重以支持进一步研究。 Releases model weights to support further research.
局限 · Limitations
仅限于 1.5B 参数模型；扩展到更大模型可能需要额外调优。 Limited to 1.5B parameter models; scaling to larger models may require additional tuning.