Efficiently acquiring external knowledge and up-to-date information is essential for effective reasoning and text generation in large language models (LLMs). Prompting advanced LLMs with reasoning capabilities to use search engines during inference is often suboptimal, as the LLM might not fully possess the capability on how to interact optimally with the search engine. This paper introduces Search-R1, an extension of reinforcement learning (RL) for reasoning frameworks where the LLM learns to autonomously generate (multiple) search queries during step-by-step reasoning with real-time retrieval. Search-R1 optimizes LLM reasoning trajectories with multi-turn search interactions, leveraging retrieved token masking for stable RL training and a simple outcome-based reward function. Experiments on seven question-answering datasets show that Search-R1 improves performance by 41% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over various RAG baselines under the same setting. This paper further provides empirical insights into RL optimization methods, LLM choices, and response length dynamics in retrieval-augmented reasoning. The code and model checkpoints are available at https://github.com/PeterGriffinJin/Search-R1.
核心贡献 · Key contributions
识别了将强化学习应用于大语言模型推理与搜索引擎调用中的挑战。 Identifies challenges of applying RL to LLM reasoning with search engine calling.
提出了 Search-R1，一种新颖的强化学习框架，支持推理与搜索交错进行，并通过词元掩码实现稳定训练。 Proposes Search-R1, a novel RL framework supporting interleaved reasoning and search with token masking for stable training.
在七个问答数据集上，相对于强基线分别实现了 26%、21%和 10%的平均相对提升。 Achieves 26%, 21%, and 10% average relative improvement over strong baselines on seven QA datasets.
提供了关于检索增强推理中强化学习方法、大语言模型选择及响应长度动态的实证见解。 Provides empirical insights on RL methods, LLM choices, and response length dynamics in retrieval-augmented reasoning.
展示了在基础模型和指令微调模型以及不同模型系列上的有效性。 Demonstrates effectiveness across base and instruction-tuned models, and different model families.
局限 · Limitations
评估仅限于七个问答数据集，未验证对其他任务的泛化性。 Evaluation limited to seven QA datasets; generalizability to other tasks not verified.
依赖单一检索器（E5）和维基百科语料库，使用其他来源时性能可能不同。 Relies on a single retriever (E5) and Wikipedia corpus; performance may vary with other sources.
使用简单的基于结果的奖励，更复杂的奖励可能进一步提升性能。 Uses simple outcome-based reward; more sophisticated rewards could further improve performance.
训练需要大量计算资源以支持带搜索交互的强化学习。 Training requires significant computational resources for RL with search interactions.
Search-R1 可能无法处理需要基于不确定性进行动态检索调整的任务。 Search-R1 may not handle tasks requiring dynamic retrieval adjustments based on uncertainty.
论文章节 · Sections（共 20）
摘要Abstract
1 引言1 Introduction
2.1 大语言模型与检索2.1 Large Language Models and Retrieval
2.2 大语言模型与强化学习2.2 Large Language Models and Reinforcement Learning
3 Search-R13 Search-R1
3.1 基于搜索引擎的强化学习3.1 Reinforcement Learning with a Search Engine
3.2 交错多轮搜索引擎调用的文本生成3.2 Text Generation with Interleaved Multi-turn Search Engine Call
3.3 训练模板3.3 Training Template
3.4 奖励建模3.4 Reward Modeling
4.1 数据集4.1 Datasets
4.2 基线方法4.2 Baselines
4.3 实验设置4.3 Experimental Setup
4.4 性能表现4.4 Performance
5.1 不同强化学习方法：PPO 与 GRPO5.1 Different RL methods: PPO vs. GRPO
5.2 基础模型与指令微调模型5.2 Base vs. Instruct LLMs
5.3 响应长度研究5.3 Response Length Study
5.4 检索令牌损失掩码研究5.4 Study of Retrieved Tokens Loss Masking