This article provides a comprehensive overview of reasoning models in the field of large language models (LLMs), defining them as systems that excel at complex, multi-step tasks such as puzzles, advanced mathematics, and coding challenges. It outlines four primary approaches to building and enhancing reasoning capabilities: inference-time scaling, pure reinforcement learning, supervised fine-tuning combined with RL, and distillation. The article uses DeepSeek's R1 pipeline as a case study, detailing how its variants (R1-Zero, R1, and R1-Distill) were developed, highlighting the surprising emergence of reasoning from pure RL without initial SFT. It also discusses the strengths and limitations of reasoning models, noting their higher cost and verbosity, and advises using them only for tasks that genuinely require complex reasoning. The conclusion emphasizes that while reasoning models are a valuable specialization, they are not a universal solution, and the choice of model should align with the task's complexity.
核心贡献 · Key contributions
将推理模型定义为专门处理复杂多步任务的 LLM，并将其与通用 LLM 区分开来。 Defines reasoning models as LLMs specialized for complex multi-step tasks, distinguishing them from general LLMs.
概述了构建推理模型的四种主要方法：推理时扩展、纯强化学习、SFT+RL 和蒸馏。 Outlines four main approaches to building reasoning models: inference-time scaling, pure RL, SFT+RL, and distillation.
强调 DeepSeek-R1-Zero 的纯强化学习方法，表明无需 SFT 即可涌现推理能力，即“顿悟”时刻。 Highlights DeepSeek-R1-Zero's pure RL approach, showing reasoning emerges without SFT, an 'Aha' moment.
表明对于小模型，蒸馏（在小模型上进行纯 SFT）比纯强化学习更有效，如 R1-Distill 所示。 Shows distillation (pure SFT on smaller models) is more effective than pure RL for small models, as in R1-Distill.
讨论了低成本替代方案，如 Sky-T1（450 美元）和 TinyZero（<30 美元），展示了经济高效的推理模型开发。 Discusses low-budget alternatives like Sky-T1 ($450) and TinyZero (<$30), demonstrating cost-effective reasoning model development.
局限 · Limitations
推理模型更昂贵、更冗长，可能对简单任务过度思考，导致错误。 Reasoning models are more expensive and verbose, and may overthink simple tasks, leading to errors.
推理时扩展增加了每次查询的成本，使大规模部署变得昂贵。 Inference-time scaling increases per-query cost, making large-scale deployment expensive.
纯强化学习可能不足以应对小模型；蒸馏更有效，但依赖于更强的教师模型。 Pure RL may not be sufficient for smaller models; distillation is more effective but depends on a stronger teacher model.
蒸馏不推动创新；它仅复制现有能力，不产生新的推理范式。 Distillation does not drive innovation; it only replicates existing capabilities without generating new reasoning paradigms.
由于 o1 的架构和训练细节未公开，直接比较 o1 和 DeepSeek-R1 很困难。 Direct comparison between o1 and DeepSeek-R1 is difficult due to undisclosed details about o1's architecture and training.
论文章节 · Sections（共 12）
构建与优化推理模型的方法与策略Methods and Strategies for Building and Refining Reasoning Models
如何定义“推理模型”？How do we define "reasoning model"?
何时应使用推理模型？When should we use reasoning models?
DeepSeek 训练流程概览A brief look at the DeepSeek training pipeline
构建和改进推理模型的四种主要方式The 4 main ways to build and improve reasoning models