Absolute Zero: Reinforced Self-play Reasoning with Zero Data
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→基于可验证奖励的强化学习（RLVR）通过直接从基于结果的奖励中学习，在增强大语言模型的推理能力方面显示出潜力。最近在零设置下工作的 RLVR 方法避免了标注推理过程，但仍依赖人工策划的问题和答案集进行训练。高质量人工样本的稀缺引发了对依赖人类监督的长期可扩展性的担忧，这在语言模型预训练领域已经显现。此外，在 AI 超越人类智能的假设未来中，人类提供的任务可能对超级智能系统的学习潜力有限。为解决这些问题，我们提出了一种新的 RLVR 范式，称为绝对零度，其中单个模型学习提出能最大化自身学习进度的任务，并通过解决这些任务来提高推理能力，无需依赖任何外部数据。在此范式下，我们引入了绝对零度推理器（AZR），该系统通过使用代码执行器来验证提出的代码推理任务和答案，从而自我进化其训练课程和推理能力，作为指导开放但基于学习的可验证奖励的统一来源。尽管完全在无外部数据的情况下训练，AZR 在编码和数学推理任务上实现了总体 SOTA 性能，优于依赖数万个领域内人工策划示例的现有零设置模型。此外，我们证明 AZR 可有效应用于不同模型规模，并与多种模型类别兼容。
Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning capabilities of large language models by learning directly from outcome-based rewards. Recent RLVR works that operate under the zero setting avoid supervision in labeling the reasoning process, but still depend on manually curated collections of questions and answers for training. The scarcity of high-quality, human-produced examples raises concerns about the long-term scalability of relying on human supervision, a challenge already evident in the domain of language model pretraining. Furthermore, in a hypothetical future where AI surpasses human intelligence, tasks provided by humans may offer limited learning potential for a superintelligent system. To address these concerns, we propose a new RLVR paradigm called Absolute Zero, in which a single model learns to propose tasks that maximize its own learning progress and improves reasoning by solving them, without relying on any external data. Under this paradigm, we introduce the Absolute Zero Reasoner (AZR), a system that self-evolves its training curriculum and reasoning ability by using a code executor to both validate proposed code reasoning tasks and verify answers, serving as an unified source of verifiable reward to guide open-ended yet grounded learning. Despite being trained entirely without external data, AZR achieves overall SOTA performance on coding and mathematical reasoning tasks, outperforming existing zero-setting models that rely on tens of thousands of in-domain human-curated examples. Furthermore, we demonstrate that AZR can be effectively applied across different model scales and is compatible with various model classes.