Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→基于可验证奖励的强化学习（RLVR）近期在提升大语言模型（LLM）的推理性能方面取得了显著成功，尤其是在数学和编程任务上。类似于传统强化学习帮助智能体探索和学习新策略，RLVR 被认为能使 LLM 持续自我改进，从而获得超越对应基座模型的新推理能力。在本研究中，我们通过系统探测不同模型家族、RL 算法以及数学、编程和视觉推理基准上 RLVR 训练模型的推理能力边界，使用大 k 值下的 pass@k 作为评估指标，对 RLVR 的当前状态进行了批判性检验。令人惊讶的是，我们发现当前的训练设置并未引发根本性的新推理模式。虽然 RLVR 训练模型在小 k（如 k=1）时优于基座模型，但当 k 较大时，基座模型获得了更高的 pass@k 分数。覆盖率和困惑度分析表明，观察到的推理能力源于基座模型并受其限制。将基座模型视为上限，我们的定量分析显示，六种流行的 RLVR 算法表现相似，且在利用基座模型潜力方面远未达到最优。相比之下，我们发现蒸馏可以从教师模型引入新的推理模式，并真正扩展模型的推理能力。总体而言，我们的发现表明，当前的 RLVR 方法尚未实现 RL 激发 LLM 真正新颖推理能力的潜力。这凸显了改进 RL 范式的必要性，例如持续扩展和多轮智能体-环境交互，以释放这一潜力。
Reinforcement Learning with Verifiable Rewards (RLVR) has recently demonstrated notable success in enhancing the reasoning performance of large language models (LLMs), particularly on mathematics and programming tasks. Similar to how traditional RL helps agents explore and learn new strategies, RLVR is believed to enable LLMs to continuously self-improve, thus acquiring novel reasoning abilities beyond those of the corresponding base models. In this study we critically examine the current state of RLVR by systematically probing the reasoning capability boundaries of RLVR-trained LLMs across various model families, RL algorithms, and math, coding, and visual reasoning benchmarks, using pass@k at large k values as the evaluation metric. Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models at small k (e.g., k = 1), the base models achieve a higher pass@k score when k is large. Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model. Treating the base model as an upper bound, our quantitative analysis shows that six popular RLVR algorithms perform similarly and remain far from optimal in leveraging the potential of the base model. By contrast, we find that distillation can introduce new reasoning patterns from the teacher and genuinely expand the model's reasoning capabilities. Overall, our findings suggest that current RLVR methods have not yet realized the potential of RL to elicit truly novel reasoning abilities in LLMs. This highlights the need for improved RL paradigms, such as continual scaling and multi-turn agent-environment interaction, to unlock this potential.