Language model pretraining with next token prediction has proved effective for scaling compute but is limited to the amount of available training data. Scaling reinforcement learning (RL) unlocks a new axis for the continued improvement of artificial intelligence, with the promise that large language models (LLMs) can scale their training data by learning to explore with rewards. However, prior published work has not produced competitive results. In light of this, we report on the training practice of Kimi k1.5, our latest multi-modal LLM trained with RL, including its RL training techniques, multi-modal data recipes, and infrastructure optimization. Long context scaling and improved policy optimization methods are key ingredients of our approach, which establishes a simplistic, effective RL framework without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models. Notably, our system achieves state-of-the-art reasoning performance across multiple benchmarks and modalities -- e.g., 77.5 on AIME, 96.2 on MATH 500, 94-th percentile on Codeforces, 74.9 on MathVista -- matching OpenAI's o1. Moreover, we present effective long2short methods that use long-CoT techniques to improve short-CoT models, yielding state-of-the-art short-CoT reasoning results -- e.g., 60.8 on AIME, 94.6 on MATH500, 47.3 on LiveCodeBench -- outperforming existing short-CoT models such as GPT-4o and Claude Sonnet 3.5 by a large margin (up to +550%).
核心贡献 · Key contributions
提出一种基于长上下文缩放和改进策略优化的简化 RL 框架，无需蒙特卡洛树搜索等复杂技术，实现最先进的推理性能。 Proposes a simplistic RL framework for LLMs using long context scaling and improved policy optimization, achieving state-of-the-art reasoning without complex techniques like Monte Carlo tree search.
引入部分回滚技术，将 RL 训练高效扩展到 128k 上下文窗口，实现随上下文长度增加的性能持续提升。 Introduces partial rollouts to efficiently scale RL training to 128k context windows, enabling continued performance improvement with longer contexts.
开发有效的长到短方法（如长度惩罚、模型合并），将长思维链推理迁移到短思维链模型，取得最先进的短思维链结果。 Develops effective long2short methods (e.g., length penalty, model merging) to transfer long-CoT reasoning to short-CoT models, achieving state-of-the-art short-CoT results.
证明上下文长度缩放是 LLM 中 RL 持续缩放的关键维度，小模型通过更长的思维链可媲美大模型。 Demonstrates that scaling context length is a key dimension for continued RL scaling with LLMs, with smaller models matching larger ones via longer CoT.
在 AIME、MATH 500、Codeforces 和 MathVista 上实现多模态最先进推理，媲美 OpenAI 的 o1。 Achieves multi-modal state-of-the-art reasoning on AIME, MATH 500, Codeforces, and MathVista, matching OpenAI's o1.
提出混合部署框架实现高效在线策略 RL 训练，将训练到推理的转换时间缩短至一分钟以内。 Presents a hybrid deployment framework for efficient on-policy RL training, reducing training-to-inference transition time to under one minute.
局限 · Limitations
简化 RL 框架可能无法泛化到需要显式搜索或价值函数的任务，超出长思维链范围。 The simplistic RL framework may not generalize to tasks requiring explicit search or value functions beyond long CoT.
长到短方法依赖高质量长思维链教师模型的可用性，并非所有领域都具备。 Long2short methods rely on the availability of a high-quality long-CoT teacher model, which may not be available for all domains.
研究聚焦于具有明确奖励的可验证问题，对主观评估的开放式任务适用性有限。 The study focuses on verifiable problems with clear rewards; applicability to open-ended tasks with subjective evaluation is limited.
将上下文长度扩展到 128k 会增加计算成本，在资源受限场景下可能不可行。 Scaling context length to 128k increases computational cost and may not be feasible for resource-constrained settings.
长到短 RL 方法仍可能在短模型中产生过度思考，因为长度惩罚预热未能完全消除。 The long2short RL method may still produce overthinking in short models, as length penalty warm-up does not fully eliminate it.
论文章节 · Sections（共 15）
摘要Abstract
1 引言1 Introduction
2 方法：基于大语言模型的强化学习2 Approach: Reinforcement Learning with LLMs
2.1 强化学习提示集整理2.1 RL Prompt Set Curation
2.2 长思维链监督微调2.2 Long-CoT Supervised Fine-Tuning
2.3 强化学习2.3 Reinforcement Learning
2.4 长转短：面向短思维链模型的上下文压缩2.4 Long2short: Context Compression for Short-CoT Models