This article analyzes DeepSeek's release of R1, an open-weights reasoning language model, and its training recipe, which replicates OpenAI's o1 approach. The core method involves a four-stage process: cold-start supervised finetuning on synthetic reasoning data, large-scale reinforcement learning (RL) on verifiable problems, rejection sampling to broaden general capabilities, and a final RL stage mixing reasoning and preference tuning. The author highlights R1-Zero, an RL-only variant trained without SFT, as a key proof that RL alone can induce reasoning behaviors, though with usability issues. The article argues that this open release marks a turning point in reasoning model research, shifting from opaque blog posts to a clear, reproducible paradigm, and predicts rapid progress and a price war in 2025. For readers, the key takeaway is that the recipe's success hinges on a strong base model, RL with verifiable rewards, and that technical innovations are not moats, as open models like R1 can match proprietary ones at a fraction of the cost.
核心贡献 · Key contributions
DeepSeek R1 是首个通过大规模强化学习训练、无需监督微调作为初始步骤的开源权重推理模型。 DeepSeek R1 is the first open-weights reasoning model trained via large-scale RL without SFT as a preliminary step.
四阶段训练方案包括冷启动监督微调、大规模强化学习、拒绝采样以及用于通用对齐的第二阶段强化学习。 The four-stage training recipe includes cold-start SFT, large-scale RL, rejection sampling, and a second RL stage for general alignment.
R1-Zero 证明在基础模型上仅使用强化学习即可引发自我验证和反思等涌现推理行为。 R1-Zero demonstrates that pure RL on a base model can elicit emergent reasoning behaviors like self-verification and reflection.
该模型以极低的推理成本（每百万输入词元 0.55 美元对比 15 美元）达到了与 OpenAI o1 相当的性能。 The model achieves competitive performance with OpenAI's o1 at a fraction of the inference cost ($0.55 vs $15 per million input tokens).
将 R1 的推理轨迹蒸馏到较小模型中可显著提升其性能，为未来的强化学习实验提供了强基线。 Distillation of R1's reasoning traces into smaller models significantly boosts their performance, providing a strong baseline for future RL experiments.
论文证实训练时强化学习缩放是推理改进的关键，而非推理时搜索方法。 The paper confirms that training-time RL scaling is the key to reasoning improvements, not inference-time search methods.
局限 · Limitations
该方法需要具有长上下文能力的极强基础模型，限制了在较小或较旧模型上的适用性。 The method requires a very strong base model with long-context capabilities, limiting applicability to smaller or older models.
R1-Zero 存在语言切换和格式错误等可靠性问题，实际使用需要额外的监督微调。 R1-Zero exhibits reliability issues like language switching and formatting errors, necessitating additional SFT for practical use.
确切的奖励设计和数据组成未完全公开，使得复现和比较变得困难。 The exact reward design and data composition are not fully disclosed, making replication and comparison difficult.
用于通用对齐的第二阶段强化学习很复杂，若未仔细平衡可能降低推理性能。 The second RL stage for general alignment is complex and may degrade reasoning performance if not carefully balanced.
蒸馏结果表明，没有大规模强化学习训练，较小模型可能无法涌现高级推理能力。 The distillation results suggest that advanced reasoning capabilities may not emerge in smaller models without large-scale RL training.
论文章节 · Sections（共 8）
是的，敲响真正的 o1 复制钟声给 DeepSeek R1 🔔🔔🔔。我们下一步走向何方。Yes, ring the true o1 replication bells for DeepSeek R1 🔔🔔🔔. Where we go next.
DeepSeek R1 推理训练方案The DeepSeek R1 training recipe for reasoning
步骤 0：训练 R1-Zero，用合成数据初始化 R1Step 0. Training R1-Zero to initialize R1 with synthetic data