This paper introduces Group Sequence Policy Optimization (GSPO), our stable, efficient, and performant reinforcement learning algorithm for training large language models. Unlike previous algorithms that adopt token-level importance ratios, GSPO defines the importance ratio based on sequence likelihood and performs sequence-level clipping, rewarding, and optimization. We demonstrate that GSPO achieves superior training efficiency and performance compared to the GRPO algorithm, notably stabilizes Mixture-of-Experts (MoE) RL training, and has the potential for simplifying the design of RL infrastructure. These merits of GSPO have contributed to the remarkable improvements in the latest Qwen3 models.
核心贡献 · Key contributions
提出组序列策略优化（GSPO），一种稳定高效的 LLM 强化学习算法。 Proposes Group Sequence Policy Optimization (GSPO), a stable and efficient RL algorithm for LLMs.
发现 GRPO 的词元级重要性权重导致高方差噪声和模型崩溃。 Identifies that GRPO's token-level importance weights cause high-variance noise and model collapse.
基于序列似然定义重要性比率，符合重要性采样原理。 Defines importance ratio based on sequence likelihood, aligning with importance sampling principles.
证明相比 GRPO 具有更优的训练稳定性和性能，尤其适用于 MoE 模型。 Demonstrates superior training stability and performance over GRPO, especially for MoE models.
消除了 MoE 强化学习训练中路由重放等复杂稳定策略的需求。 Eliminates need for complex stabilization strategies like Routing Replay in MoE RL training.
为最新 Qwen3 模型的显著改进做出了贡献。 Contributes to significant improvements in the latest Qwen3 models.
局限 · Limitations
评估仅限于特定基准（AIME、LiveCodeBench、CodeForces）。 Evaluation is limited to specific benchmarks (AIME, LiveCodeBench, CodeForces).
实验聚焦于 Qwen3-30B-A3B-Base 模型，未展示对其他架构的泛化性。 Experiments focus on Qwen3-30B-A3B-Base model; generalizability to other architectures is not shown.
引入了词元级变体 GSPO-token，但未进行广泛评估。 Token-level variant GSPO-token is introduced but not extensively evaluated.
未提供梯度方差减少的理论分析。 Theoretical analysis of gradient variance reduction is not provided.
讨论了推理引擎精度等实际部署问题，但未经验证。 Practical deployment considerations like inference engine precision are discussed but not empirically validated.
论文章节 · Sections（共 12）
摘要Abstract
1 引言1 Introduction
2 预备知识2 Preliminaries
3 动机3 Motivation
4.1 GSPO：组序列策略优化4.1 GSPO: Group Sequence Policy Optimization
4.2 梯度分析4.2 Gradient Analysis
4.3 GSPO-token：一种词元级目标变体4.3 GSPO-token: A Token-level Objective Variant
5.1 实验结果5.1 Empirical Results
5.2 关于裁剪比例的奇特观察5.2 Curious Observation on Clipping Fractions
5.3 GSPO 对 MoE 训练的益处5.3 Benefit of GSPO for MoE Training
5.4 GSPO 对 RL 基础设施的益处5.4 Benefit of GSPO for RL Infrastructure