MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement
打开互动全文版（逐段中英对照 + 图/公式 + 针对本篇提问）→强化学习（RL）是推动大型基础模型走向自我改进的核心训练范式。本报告介绍 MiMo-V2.6 系列——一个通过扩展 RL 算力来拓展模型智能边界的全模态模型家族。在 RL 之前，我们在广泛的多模态语料上进行中期训练，以提供充足的探索空间，并在预训练的混合 SWA 架构上构建坚实基础，以支撑后续的规模扩展。我们从三个维度扩展 RL 算力：（1）更大的批次与更高的吞吐量，采用异步训练，在最长 1M 的上下文长度下每步消耗 1,568 个样本和 2.7–3.7B token；（2）更多样、更复杂的环境，在多种智能体框架下覆盖代码、通用、视觉与网络领域；（3）更多评分算力，通过分组式智能体评分，为长程任务提供更准确的奖励信号，并引导模型给出更短、更省 token 的解法。为在规模化下保持训练稳定，我们冻结 MoE 路由器，并建立多层防御以抵御奖励黑客。我们还构建了面向混合任务智能体 RL 的基础设施，包括统一的轨迹表示、高并发多框架 rollout、控制面与数据面解耦，以及训练-推理一致性。我们开源了训练动态、RL 环境与 RL 框架，以促进复现及关于规模化 RL 与模型自我改进的进一步研究。
Reinforcement learning (RL) is the central training paradigm for advancing large foundation models towards self-improvement. This report introduces the MiMo-V2.6 series, an omni-modal family that pushes the frontier of model intelligence by scaling RL compute. Prior to RL, we conduct mid-training on a broad multimodal corpus to provide ample exploration space, and build a solid infrastructure on the pretrained hybrid-SWA architecture to support subsequent scale-up. We scale RL compute along three dimensions: (1) larger batches and higher throughput, with an asynchronous training that consumes 1,568 samples and 2.7-3.7B tokens per step at context lengths of up to 1M; (2) more diverse and complex environments, spanning code, general, visual, and cyber domains under a mixture of agent harnesses; and (3) more grader compute, via groupwise agentic grading that yields more accurate reward signals for long-horizon tasks and steers the model towards shorter, more token-efficient solutions. To keep training stable at scale, we freeze the MoE router and establish a multi-layer defense against reward hacking. We further build infrastructure for mixed-task agentic RL, including a unified trajectory representation, high-concurrency multi-framework rollout, decoupled control and data planes, and training-inference consistency. We open-source the training dynamics, RL environments, and RL framework to facilitate reproduction and further research on scaled RL and model self-improvement.
强化学习（RL）是推动大型基础模型迈向自我改进的核心训练范式。本报告介绍 MiMo-V2.6 系列——一个全模态模型家族，通过扩展 RL 算力来推进模型智能的前沿。在 RL 之前，我们在广泛的多模态语料上进行中期训练，以提供充足的探索空间，并在预训练的 hybrid-SWA 架构上构建坚实的基础设施，以支撑后续的规模扩展。我们从三个维度扩展 RL 算力：（1）更大的批次与更高的吞吐量，采用异步训练，在上下文窗口长度最高达 1M 时，每步消耗 1,568 个样本和 2.7 \(\sim\) 3.7B 个 token；（2）更多样、更复杂的环境，在混合智能体 harness 下覆盖代码、通用、视觉和网络领域；（3）更多的评分器算力，通过分组式智能体评分，为长时程任务产生更准确的奖励信号，并引导模型给出更短、更 token 高效的解。为在规模扩展下保持训练稳定，我们冻结 MoE 路由器，并建立多层防御以抵御奖励黑客（reward hacking）。我们进一步构建了面向混合任务智能体式 RL 的基础设施，包括统一的轨迹表示、高并发多框架 rollout、解耦的控制面与数据面，以及训练–推理一致性。我们开源了训练动态、RL 环境和 RL 框架，以促进对扩展 RL 与模型自我改进的复现和进一步研究。
Reinforcement learning (RL) is the central training paradigm for advancing large foundation models towards self-improvement. This report introduces the MiMo-V2.6 series, an omni-modal family that pushes the frontier of model intelligence by scaling RL compute. Prior to RL, we conduct mid-training on a broad multimodal corpus to provide ample exploration space, and build a solid infrastructure on the pretrained hybrid-SWA architecture to support subsequent scale-up. We scale RL compute along three dimensions: (1) larger batches and higher throughput, with an asynchronous training that consumes 1,568 samples and 2.7 \(\sim\) 3.7B tokens per step at context lengths of up to 1M; (2) more diverse and complex environments, spanning code, general, visual, and cyber domains under a mixture of agent harnesses; and (3) more grader compute, via groupwise agentic grading that yields more accurate reward signals for long-horizon tasks and steers the model towards shorter, more token-efficient solutions. To keep training stable at scale, we freeze the MoE router and establish a multi-layer defense against reward hacking. We further build infrastructure for mixed-task agentic RL, including a unified trajectory representation, high-concurrency multi-framework rollout, decoupled control and data planes, and training–inference consistency. We open-source the training dynamics, RL environments, and RL framework to facilitate reproduction and further research on scaled RL and model self-improvement.
递归自我改进（RSI）设想模型通过持续探索和反馈来扩展其能力。实现这一愿景需要智能体，它将模型与交互式环境耦合，从而提供自我改进所依赖的多步轨迹和反馈信号。为此，在复杂智能体任务上扩展强化学习（RL）开辟了一条具体路径。然而，实现这一潜力面临两大挑战。首先，面向基础模型的 RL 需要合适的模型架构和足够丰富的智能体探索空间。其次，扩展 RL 需要针对基础设施、环境和评分器的复杂解决方案。在本报告中，我们介绍 MiMo-V2.6 系列，包括 MiMo-V2.6-Pro（一个 1.02T 参数的混合专家模型，具有 42B 激活参数）和 MiMo-V2.6-Flash（一个 310B 参数的混合专家模型，具有 15B 激活参数），以弥合这些差距，在规模化 RL 方面迈出实践性的一步。
Recursive self-improvement (RSI) envisions models that expand their capabilities through sustained exploration and feedback. Realizing this vision requires agents, which couple models with interactive environments and thereby supply the multi-step trajectories and feedback signals that self-improvement depends on. Towards this goal, scaling reinforcement learning (RL) on complex agentic tasks thus opens a concrete path. However, realizing this potential faces two major challenges. First, RL for foundation models requires suitable model architectures and a sufficiently rich exploration space for agents. Second, scaling RL requires sophisticated solutions for infrastructure, environments, and graders. In this report, we introduce the MiMo-V2.6 series, including MiMo-V2.6-Pro, a 1.02T-parameter Mixture-of-Experts model with 42B active parameters, and MiMo-V2.6-Flash, a 310B-parameter Mixture-of-Experts model with 15B active parameters, to bridge these gaps, taking a practical step in large-scale RL.
MiMo-V2.6 的架构、预训练和中训练共同构建了一个强大且高效的基础模型。出于在低计算成本下保留全局上下文的需求，MiMo-V2.6 构建了一个混合稀疏 MoE Transformer 主干，将局部滑动窗口注意力（SWA）与全局注意力（GA）交错，并辅以轻量级视觉编码器和音频编码器。此外，大规模预训练赋予模型广泛的知识以及跨文本、音频和视频的多功能多模态理解能力。我们进一步引入以智能体为中心的中训练阶段，扩展智能体任务的探索空间，使模型能够在后训练期间发现更有效的任务解决轨迹。
The architecture, pre-training, and mid-training of MiMo-V2.6 jointly establish a powerful and efficient foundation model. Motivated by the need to preserve global context at low computational cost, MiMo-V2.6 builds a hybrid sparse MoE Transformer backbone that interleaves Local Sliding Window Attention (SWA) with Global Attention (GA), augmented by a lightweight visual encoder and audio encoders. In addition, large-scale pre-training equips the model with extensive knowledge and versatile multimodal understanding across text, audio, and video. We further introduce an agent-centric mid-training phase that expands the exploration space for agentic tasks, enabling the model to discover more effective task-solving trajectories during post-training.
接下来，我们通过一个系统性的、协同设计的框架来扩展 RL 算力，该框架涵盖三个关键组件。首先，我们通过完全异步训练来扩展批量大小和训练吞吐量，在高达 1M 的上下文长度下，每步处理数千个长时程轨迹和数十亿个 token。其次，我们扩展 RL 环境的多样性和复杂性，涵盖编码、通用、视觉和网络安全任务，使用多样化的智能体工具架并加强对奖励黑客的防护。改变工具架和任务都能提高模型的泛化能力。第三，我们引入分组智能体评分，以提供超越二元测试用例的更具信息量的奖励信号。我们不是为所有通过测试用例的解决方案分配相同的奖励，而是在每个组内比较解决方案，以区分问题解决质量和行为。通过我们提出的分组奖励合成（GRS）和分组优势重分配（GAR），这些细粒度的区分被转化为更具信息量的学习信号，引导模型走向更准确和高效的解决方案。
Next, we scale the RL compute through a systematic, co-designed framework spanning three key components. First, we scale batch size and training throughput through fully asynchronous training, processing thousands of long-horizon rollouts and billions of tokens per step at context lengths of up to 1M. Second, we scale the diversity and complexity of RL environments across coding, general, visual, and cybersecurity tasks, using diverse agent harnesses and strengthening safeguards against reward hacking. Varying both the harness and the task improves the model generalization. Third, we introduce groupwise agentic grading to provide more informative reward signals beyond binary test cases. Instead of assigning the same reward to all solutions that pass the test cases, we compare solutions within each group to distinguish problem-solving quality and behaviors. Through our proposed Groupwise Reward Synthesis (GRS) and Groupwise Advantage Redistribution (GAR), these fine-grained distinctions are converted into more informative learning signals, guiding the model toward more accurate and efficient solutions.
大规模实现这些对基础设施提出了研究和工程挑战。为了支持灵活的智能体交互场景，我们设计了一种统一的轨迹表示和一种惩罚机制，共同细化学习信号。为了处理大型训练批次，我们通过一个工具架池在多样化的智能体框架之间维持高并发交互，并将控制平面与数据平面解耦，以缓冲和传输携带路由和多模态有效载荷的大规模轨迹。一种样本混合机制，与动态采样和部分展开协同工作，稳定训练批次中每个任务的样本组成。我们在 MoE 路由和 top-p 采样候选集上对齐训练和推理引擎，并针对 RL 工作负载优化两个引擎。
Realizing these at scale presents both research and engineering challenges for infrastructure. To support flexible agentic interaction scenarios, we design a unified trajectory representation and a penalty mechanism that together refine the learning signal. To handle large training batches, we sustain high-concurrency interaction across diverse agent frameworks through a pool of harnesses, and decouple the control plane from the data plane to buffer and transfer massive trajectories carrying routing and multi-modal payloads. A sample mixing mechanism, working together with dynamic sampling and partial rollout, stabilizes per-task sample composition in train batches. We align the training and inference engines on MoE routing and top-p sampling candidate sets, and optimize both engines for RL workloads.
扩展 RL 算力显著释放了模型在可验证任务（如编码）和较难验证任务（如网页开发）上的潜力。如图 1 所示，随着训练步数增加，MiMo-V2.6-Pro 和 MiMo-V2.6-Flash 在报告的基准测试中均稳步提升性能。此外，这些增益不仅限于单一领域，而是在多种任务中持续存在，涵盖 DeepSWE 上的编码、AutomationBench 上的通用工作流、MiMo Visual Coding 上的视觉任务以及 MiMo Cyber Bench 上的网络安全。这些增益表明，MiMo-V2.6 能够在日常协作场景中实现更强的复杂问题解决和更可靠的性能，反映出更通用的智能体智能。
Scaling RL compute substantially unlocks model potential on both verifiable tasks such as coding and less verifiable tasks such as web development. As shown in Figure 1, both MiMo-V2.6-Pro and MiMo-V2.6-Flash steadily improve their performance across the reported benchmarks as training steps increase. Moreover, these gains are not confined to a single domain, but hold consistently across diverse tasks spanning coding on DeepSWE, general workflows on AutomationBench, visual tasks on MiMo Visual Coding, and cybersecurity on MiMo Cyber Bench. These gains indicate that MiMo-V2.6 is capable of stronger complex problem solving and more reliable performance in daily co-work scenarios, reflecting more general agentic intelligence.
为促进智能体式强化学习（agentic RL）研究，我们开源了一套覆盖强化学习开发栈关键组件的综合套件，包括轻量级模型 MiMo-V2.6-Distill-Qwen-9B、跨多个领域且带有验证器的精选任务环境、端到端强化学习训练框架，以及用于灵活智能体式交互的可组合迷你 harness。这些组件共同提供了一个统一且易用的平台，用于在多样化任务、环境和智能体配置下研究智能体式强化学习，同时降低了复现和扩展大规模强化学习实验的工程门槛。基于 MiMo-V2.6-Distill-Qwen-9B 的实验表明，在不同任务和智能体 harness 上均能获得一致的强化学习增益，验证了所提出套件的有效性与通用性。我们希望这套开源套件能够成为社区公平、强大且可复现的基线，支持对智能体式强化学习进行系统性研究，并推动研究迈向递归自我改进。
To promote research in agentic RL, we open-source a comprehensive suite covering the key components of the RL development stack, including the lightweight model MiMo-V2.6-Distill-Qwen-9B, curated task environments with verifiers across multiple domains, an end-to-end RL training framework, and a composable mini-harness for flexible agentic interactions. Together, these components provide a unified and accessible platform for studying agentic RL under diverse tasks, environments, and agent configurations, while reducing the engineering barriers to reproducing and extending large-scale RL experiments. Experiments with MiMo-V2.6-Distill-Qwen-9B demonstrate consistent RL gains across diverse tasks and agent harnesses, validating the effectiveness and generality of the proposed suite. We hope this open-source suite can serve as a fair, strong, and reproducible baseline for the community, enabling systematic investigation of agentic RL and advancing research toward recursive self-improvement.
如图 2 所示，MiMo-V2.6 采用标准的 Transformer 主干，并通过轻量级投影器连接视觉与音频编码器进行增强。
As illustrated in Figure 2, MiMo-V2.6 follows a standard Transformer backbone, augmented with visual and audio encoders connected through lightweight projectors.
MiMo-V2.6 的文本主干主要由重复的混合块构成，这些混合块交替使用局部滑动窗口注意力机制（SWA）与全局注意力机制（GA）。它堆叠了 \(M\) 个混合块，每个混合块由 \(N\) 个连续的 SWA 块后接一个 GA 块组成。唯一的例外是第一个 Transformer 块，它使用带稠密前馈网络（FFN）的全局注意力机制，以稳定早期的表征学习。MiMo-V2.6 中使用的滑动窗口大小 \(W\) 为 \(128\)。SWA 块与 GA 块均采用不含共享专家的稀疏 MoE FFN。MiMo-V2.6 还集成了基于 SWA 与稠密 FFN 的 MTP，以在预训练期间提升模型性能。关于文本主干架构的更全面描述可见于。模型配置如表 1 所示。
The text backbone of MiMo-V2.6 is mainly composed of repeated hybrid blocks that interleave Local Sliding Window Attention (SWA) and Global Attention (GA). It stacks \(M\) hybrid blocks, each structured with \(N\) consecutive SWA blocks followed by a GA block. The only exception is the very first Transformer block, which uses global attention with a dense Feed-Forward Network (FFN) to stabilize early representation learning. The sliding window size \(W\) used in MiMo-V2.6 is \(128\). Both the SWA block and the GA block utilize a sparse MoE FFN without shared experts. MiMo-V2.6 also integrates an SWA and dense FFN based MTP to improve model performance during pre-training. A more comprehensive description of the text backbone architecture can be found in . The model configurations are shown in Table 1.
MiMo-ViT 采用混合注意力架构。它将 MiMo-VL-7B 中固定的、不重叠的窗口注意力机制替换为带 sink 增强的 SWA，使信息能够在连续层之间跨窗口边界交换，从而缓解视觉碎片化问题。局部 SWA 层在行优先与列优先的 token 序列化方式之间交替，以支持信息沿两个空间轴传播。最后，周期性地插入 GA 层以直接聚合全局上下文。该设计大幅降低了高分辨率视觉处理的计算开销，同时达到与 GA ViT 相当的性能。更多分析请参见[附录]。
MiMo-ViT adopts a hybrid attention architecture. It replaces the fixed, non-overlapping window attention in MiMo-VL-7B with sink-augmented SWA, enabling information exchange across window boundaries over successive layers and mitigating visual fragmentation. Local SWA layers alternate between row-major and column-major token serialization to support information propagation along both spatial axes. Finally, GA layers are inserted periodically to aggregate global context directly. This design substantially reduces the computational cost of high-resolution visual processing while achieving performance comparable to GA ViTs. Please refer to [Appendix] for more analysis.
MiMo-V2.6 中的音频编码包含两个阶段：首先使用 Audio Tokenizer 进行音频 tokenization，随后进行 patch encoding。Audio Tokenizer 编码器首先通过一个两层卷积前端处理 log-mel 频谱图，该前端将帧率减半。随后，得到的序列被送入一个采用因果混合注意力架构的 Transformer，该架构交替使用 SWA 层和 GA 层。SWA 层捕获局部时间依赖关系，而 GA 层则聚合整个前序序列的上下文。随后的下采样卷积进一步将帧率减半至 \(25\) Hz，之后一个 20 层残差向量量化器（RVQ）将每一帧表示为 \(20\) 个离散音频 token。Audio Tokenizer 遵循 MiMo-Audio 的训练配方，并在涵盖语音、音乐和其他音频内容的 2000 万小时音频上进行训练。
MiMo-V2.6 中的音频编码包含两个阶段：首先使用 Audio Tokenizer 进行音频 tokenization，随后进行 patch encoding。Audio Tokenizer 编码器首先通过一个两层卷积前端处理 log-mel 频谱图，该前端将帧率减半。随后，得到的序列被送入一个采用因果混合注意力架构的 Transformer，该架构交替使用 SWA 层和 GA 层。SWA 层捕获局部时间依赖关系，而 GA 层则聚合整个前序序列的上下文。随后的下采样卷积进一步将帧率减半至 \(25\) Hz，之后一个 20 层残差向量量化器（RVQ）将每一帧表示为 \(20\) 个离散音频 token。Audio Tokenizer 遵循 MiMo-Audio 的训练配方，并在涵盖语音、音乐和其他音频内容的 2000 万小时音频上进行训练。
音频 patch 编码器遵循 MiMo-Audio 的架构设计，并与文本主干联合训练。在每个时间步，音频 token 使用独立的嵌入表进行嵌入，每个 RVQ 码本对应一个嵌入表，然后将得到的嵌入求和以形成单帧表示。每四个连续帧被分组为一个音频 patch，并由一个 Transformer 处理，该 Transformer 的注意力机制为限制在该 patch 内的双向自注意力。然后，四个输出表示被拼接并线性投影为单个主干输入嵌入，从而将音频序列速率从 \(25\) Hz 降低到 \(6.25\) Hz。
音频 patch 编码器遵循 MiMo-Audio 的架构设计，并与文本主干联合训练。在每个时间步，音频 token 使用独立的嵌入表进行嵌入，每个 RVQ 码本对应一个嵌入表，然后将得到的嵌入求和以形成单帧表示。每四个连续帧被分组为一个音频 patch，并由一个 Transformer 处理，该 Transformer 的注意力机制为限制在该 patch 内的双向自注意力。然后，四个输出表示被拼接并线性投影为单个主干输入嵌入，从而将音频序列速率从 \(25\) Hz 降低到 \(6.25\) Hz。
MiMo-V2.6 中的投机解码采用一个多词预测（MTP）模块，其设计遵循 DFlash 的块扩散（block diffusion）方案。该起草器由 5 层 Transformer 组成，并配备稠密前馈网络。它以主干网络的隐藏特征和一个干净的锚点词元为条件，在单次前向传播中预测后续 7 个词元，供主干网络并行验证。所有起草层均使用带分组查询的滑动窗口注意力机制（SWA），以限制注意力计算量和 KV 缓存大小。词元在其起草块内进行双向注意力计算，并且最多关注锚点之前的 \(1{,}024\) 个主干上下文位置。
Speculative decoding in MiMo-V2.6 uses a multi-token prediction (MTP) module following the block diffusion design of DFlash. The drafter comprises 5 Transformer layers with dense feed-forward networks. Conditioned on backbone hidden features and a clean anchor token, it predicts 7 subsequent tokens in a single forward pass for parallel verification by the backbone. All draft layers use sliding-window attention (SWA) with grouped queries to limit attention computation and KV-cache size. Tokens attend bidirectionally within their draft block and to at most \(1{,}024\) backbone context positions preceding the anchor.
MiMo-V2.6 的预训练语料涵盖文本、视觉与音频。文本语料来源广泛，包括公开网页内容、书籍、学术论文、代码以及 STEM 材料。视觉语料包括图像描述、定位（grounding）、OCR、GUI、对话、视频以及视觉编程数据。在音频方面，我们大规模构建了多样化的高质量数据，并将数据组织为三种任务格式：语音-文本交错、自动语音识别（ASR）以及通用音频描述。
The pre-training corpus of MiMo-V2.6 spans text, vision, and audio. The text corpus draws from a broad range of sources, including public web content, books, academic papers, code, and STEM materials. The vision corpus includes image-captioning, grounding, OCR, GUI, conversation, video, and visual-coding data. For audio, we curate diverse, high-quality data at scale and organize the data into three task formats: speech-text interleaving, automatic speech recognition (ASR), and general audio captioning.
我们采用两阶段预训练策略。第一阶段，我们在纯文本数据上训练语言主干，以建立强大的基础语言能力。第二阶段，我们将主干与自研的预训练 ViT 和音频编码器整合，并在全模态数据上联合训练完整模型，使其能够理解图像、视频与音频。
We adopt a two-stage pre-training strategy. In the first stage, we train the language backbone on text-only data to establish strong foundational language capabilities. In the second stage, we integrate the backbone with our in-house pre-trained ViT and audio encoder and jointly train the full model on omni-modal data, enabling it to understand images, videos, and audio.
我们以 32K token 的上下文窗口开始预训练，并在训练中途将其扩展至 256K。MiMo-V2.6-Flash 在 48T token 上训练，其中文本阶段 26T token、全模态阶段 22T token。MiMo-V2.6-Pro 在 30T token 上训练，两个阶段分别为 27T 和 3T token。我们在 MiMo-V2.6 的预训练中采用 AdamW 优化器。
We begin pre-training with a context length of 32K tokens and extend it to 256K partway through training. MiMo-V2.6-Flash is trained on 48T tokens, comprising 26T tokens in the text stage and 22T in the omni stage. MiMo-V2.6-Pro is trained on 30T tokens, with 27T and 3T tokens in the two stages, respectively. We employ the AdamW optimizer in the pre-training of MiMo-V2.6.
预训练在文本、视觉和音频上构建广泛的知识与通用理解。中训练则连接这一通用基础与后续的大规模强化学习（RL）：进一步发展模型的智能体式能力、扩展长上下文支持，并为大 batch 训练调整优化设置。
Pre-training builds broad knowledge and general understanding across text, vision, and audio. Mid-training bridges this general foundation and subsequent large-scale RL by further developing the model's agentic abilities, extending long-context support, and adapting the optimization setup for large-batch training.
为此，我们在一个多样、以智能体为中心的数据混合上训练 MiMo-V2.6 系列模型。其多样性同时覆盖任务领域与数据模态，将编码、通用、视觉和研究任务中的真实智能体轨迹，与高质量文本、仓库级代码以及图像、视频和音频数据相结合。训练分两个阶段进行：我们首先以 256K 上下文窗口训练，并将大部分算力分配给该阶段；随后在最后阶段将上下文窗口扩展至 1M。
To this end, we train the MiMo-V2.6 model series on a diverse, agent-centric data mixture. Its diversity spans both task domains and data modalities, combining realistic agent trajectories across coding, general, visual, and research tasks with high-quality text, repository-level code, and image, video, and audio data. Training proceeds in two stages: we first train with a 256K context length, allocating the majority of compute to this stage, and then extend the context length to 1M in the final stage.
我们的基座模型使用 AdamW 预训练，但初步实验表明，在混合任务强化学习中，随着 batch size 增大，优化效率会逐渐下降。与逐元素自适应调整参数的 AdamW 不同，Muon 通过将隐藏权重的更新正交化来利用其矩阵结构。这种基于矩阵的更新在超过临界 batch size 后仍保持更强的数据效率，使 Muon 对大规模 batch 训练尤为有吸引力。因此，在中训练期间，我们对隐藏权重矩阵改用 Muon 优化器的一个变体 Muown，为模型后续的大 batch 强化学习做准备。Muown 在 Muon 基础上增加了显式的行范数控制，以极小的开销缓解谱范数漂移，并降低对权重衰减的敏感性。Embedding、LM head 和 MoE router 仍继续使用 AdamW。
Our base model was pre-trained with AdamW, but our preliminary experiments showed diminishing optimization efficiency as the batch size increased in mixed-task RL. Unlike AdamW, which adapts parameters element-wise, Muon leverages the matrix structure of hidden weights by orthogonalizing their updates. This matrix-based update retains stronger data efficiency beyond the critical-batch-size regime, making Muon particularly attractive for large-batch training. We therefore switch to a variant of the Muon optimizer, Muown, for the hidden weight matrices during mid-training to prepare the model for subsequent large-batch RL. Muown augments Muon with explicit row-norm control, mitigating spectral-norm drift and reducing sensitivity to weight decay at negligible overhead. Embeddings, the LM head, and the MoE router continue to use AdamW.
已有工作报道，将经 Adam 预训练的模型切换为 Muon 训练可能导致优化器不匹配和性能下降。然而，在我们的中训练过程中，整个训练过程未观察到 loss 尖峰。
Prior work reports that switching an Adam-pre-trained model to Muon training can cause optimizer mismatch and degraded performance. Nevertheless, during our mid-training, we observe no loss spike throughout the training process.
我们在中训练期间采用 MXFP4 量化感知训练（QAT），使模型能够适应低精度计算，同时保持模型质量。
We employ MXFP4 quantization-aware training (QAT) during mid-training, allowing the model to adapt to low-precision computation while preserving model quality.
在本次发布中，我们继续拓展后训练算法的边界，尤其聚焦于扩展强化学习的算力。在短暂的监督微调（SFT）阶段之后，我们沿三个维度扩展强化学习：训练算力（§4.1）、环境与智能体运行框架（§4.2）以及评分器算力（§4.3），从而实现模型能力的根本性跃升。
In this release, we continue to push the boundaries of post-training algorithms, with a particular focus on scaling RL compute. Following a short supervised fine-tuning (SFT) stage, we scale RL along three dimensions: training computation (§4.1), environments and agent harnesses (§4.2), and grader computation (§4.3), achieving a fundamental leap in model capabilities.
我们在单次运行中跨数千个 GPU 扩展强化学习计算，为 MiMo-V2.6-Pro 和 MiMo-V2.6-Flash 的强化学习后训练分别花费 260 万美元和 90 万美元。我们使用大批次训练：1,568 个提示，组大小 \(G{=}16\)，因此每一步推出 25K 条序列，总计 27 亿至 37 亿个训练 token（每条序列约 11 万至 15 万 token）。图 3 的左图展示了扩展强化学习计算的好处。两个模型在 DeepSWE 基准上的平均@3 分数随累积成本稳步上升：在强化学习过程中，MiMo-V2.6-Pro 从 \(58.4\) 提升到 \(72.6\)，MiMo-V2.6-Flash 从 \(48.7\) 提升到 \(65.7\)。计算分为三部分：推出、评分和训练。这种扩展需要坚实的基础设施来保持训练高效稳定。
We scale RL computation across thousands of GPUs in a single run, spending $2.6M and $0.9M on RL post-training for MiMo-V2.6-Pro and MiMo-V2.6-Flash, respectively. We use a large training batch: 1,568 prompts with group size \(G{=}16\), so each step rolls out 25K sequences that amount to 2.7B–3.7B training tokens (roughly 110K–150K tokens per sequence). The left panel of Figure 3 shows the benefit of scaling RL computation. The average@3 score on the DeepSWE benchmark rises steadily with cumulative cost for both models: over the course of RL, MiMo-V2.6-Pro improves from \(58.4\) to \(72.6\) and MiMo-V2.6-Flash from \(48.7\) to \(65.7\). The computation splits into three parts: rollout, grading, and training. Such scaling calls for solid infrastructure to keep training efficient and stable.
我们如下构建强化学习目标：
We form the RL learning objective as follows:
其中 \(r\) 表示重要性采样比，\(M\) 是 token 级掩码，\(A\) 是优势。计算可以分解为三部分。第一部分是推出：对于从任务数据集并集 \(\bigcup_{d}\mathcal{D}_{d}\) 中抽取的每个提示 \(q\)，推出策略 \(\mu_{\theta_{\mathrm{old}}}\) 生成一组 \(G\) 个候选解 \(\{o_{i}\}\)。第二部分是评分：我们在智能体式评估中投入额外计算，以区分有效解决方案和行为与有缺陷的，超越二元测试结果所揭示的。这些判断为优势 \(A_{i}\) 提供信息，从而在轨迹间实现更准确的信用分配。第三部分是训练：收集的 token 通过 \(\log\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})\) 的梯度更新 \(\theta\)，并采用提示均值聚合。图 3 的右图按计算去向分解了 MiMo-V2.6-Pro 的成本：推出消耗 43.8%，训练消耗 43.5%，而评分器占剩余的 12.7%。大批次在一次运行中从跨各种环境和工具架的多样化强化学习任务中抽取（第 4.2 节）。我们还扩展评分器以提供细粒度的学习信号（第 4.3 节）。
where \(r\) denotes the importance sampling ratio, \(M\) is the token-level mask, and \(A\) is the advantage. The computation can be decomposed into three parts. The first is rollout: for every prompt \(q\) drawn from the union of task datasets \(\bigcup_{d}\mathcal{D}_{d}\), the rollout policy \(\mu_{\theta_{\mathrm{old}}}\) generates a group of \(G\) candidate solutions \(\{o_{i}\}\). The second is grading: we invest additional compute in agentic evaluation to distinguish effective solutions and behaviors from flawed ones, beyond what binary test outcomes reveal. These judgments inform the advantages \(A_{i}\), enabling more accurate credit assignment across trajectories. The third is training: the collected tokens update \(\theta\) through the gradient of \(\log\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})\), with prompt-mean aggregation. The right panel of Figure 3 breaks down the cost of MiMo-V2.6-Pro by where the compute goes: rollout consumes 43.8% and training 43.5%, while the grader takes the remaining 12.7%. A large batch draws from diverse RL tasks spanning various environments and harnesses within one run (Section 4.2). We also scale the grader to provide fine-grained learning signals (Section 4.3).
大批次有利于横向扩展 GPU：推出在序列上并行化，训练在数据并行等级上分片批次，因此吞吐量随 GPU 数量增长。由于 HBM 主要被运行批次占用，我们可以保持高算术强度以在推出中充分利用 GPU。由于推出和评分具有长尾，我们使用部分推出来保持运行批次饱和：当收集训练批次时，我们让进行中的序列中断，并在下一个推出阶段恢复它们。代价是重新预填充：每次策略更新后，延续必须重建其 KV 缓存，因此批次大小和部分推出一起选择——大批次摊销重新预填充。为了支持稳定的强化学习训练，训练-推理一致性由 R3 和 top-p 采样候选集的重放来确保。为了确保所有样本以有效梯度贡献训练，我们加入动态采样器来过滤全通过或全失败的组。在多任务强化学习中，我们实现样本混合器，以在给定不同任务的推出时长和通过率的情况下确保高效稳定的训练批次分布。为了将推出轨迹打包成大批次训练（27 亿至 37 亿 token），我们解耦数据平面和控制平面。这些设计背后的基础设施在第 6 节中描述。
A large batch is good for scaling out GPUs: rollout parallelizes over sequences and training shards the batch across data-parallel ranks, so throughput grows with the GPU count. With HBM mostly occupied by the running batch, we can keep a high arithmetic intensity to fully use the GPU in rollout. Since rollout and grading have a long tail, we use partial rollout to keep the running batch saturated: when we collect a training batch, we leave the in-flight sequences interrupted and resume them in the next rollout phase. The cost is re-prefill: a continuation must rebuild its KV cache after each policy update, so batch size and partial rollout are chosen together—a large batch amortizes the re-prefill. To support stable RL training, train–inference consistency is ensured by R3 and the replay of top-p sampling candidate sets. To ensure that all samples contribute to training with effective gradients, we incorporate a dynamic sampler to filter groups that are all-pass or all-fail. In multi-task RL, we implement a Sample Mixer to ensure efficient and stable training batch distribution given various rollout durations and pass rates of different tasks. To pack the trajectories from rollout into large training batches (2.7B–3.7B tokens), we decouple the data plane and control plane. The infrastructure behind these designs is described in Section 6.
扩展智能体式强化学习需要多样化的环境：这些环境应反映真实世界任务、支持在训练规模下可复现地执行，并提供与任务目标一致奖励。同时满足这些要求颇具挑战：真实工作流涉及异构工具与复杂状态，而自动评分器可能不完整、不一致，或易被利用。因此，我们在编码（§4.2.1）、通用专业工作流（§4.2.2）、视觉产物（§4.2.3）和网络安全（§4.2.4）等领域投入大规模环境合成与整理，并强调监督质量。本小节介绍我们如何在上述领域构建任务与环境、智能体与之交互的执行框架（§4.2.5），以及用于在训练前和训练中提升奖励可靠性、缓解奖励黑客行为的执行检查、基于 rollout 的审计和对抗性测试（§4.2.6）。
Scaling agentic RL requires diverse environments that reflect real-world tasks, support reproducible execution at training scale, and provide rewards aligned with task objectives. Meeting these requirements jointly is challenging: realistic workflows involve heterogeneous tools and complex state, while automated graders may be incomplete, inconsistent, or vulnerable to exploitation. We therefore invest in large-scale environment synthesis and curation across coding (§4.2.1), general professional workflows (§4.2.2), visual artifacts (§4.2.3), and cybersecurity (§4.2.4), emphasizing supervision quality. This subsection describes how we construct tasks and environments across these domains, the harnesses through which agents interact with them (§4.2.5), and the execution checks, rollout-based audits, and adversarial testing used to improve reward reliability and mitigate reward hacking before and during training (§4.2.6).
编码任务特别适合强化学习（RL），因为候选解决方案可以通过自动化测试执行和评估，从而大规模提供反馈。然而，仅靠可执行评估并不能保证可靠的监督：测试可能忽略所需行为、拒绝有效实现，或在多次执行中产生不一致的结果。因此，核心挑战在于扩展任务构建规模，同时确保奖励忠实反映任务正确性。为此，我们投入大量精力构建可扩展的编码智能体任务合成与整理流程，重点在于监督的准确性和鲁棒性。图 4 总结了任务合成路径以及用于评估监督准确性和鲁棒性的检查。
Coding tasks are particularly well suited to RL, as candidate solutions can be executed and evaluated through automated tests, providing feedback at scale. However, executable evaluation alone does not guarantee reliable supervision: tests may overlook required behavior, reject valid implementations, or yield inconsistent outcomes across executions. The central challenge is therefore to scale task construction while ensuring that rewards faithfully reflect task correctness. Motivated by this, we devote substantial effort to building a scalable synthesis and curation pipeline for coding-agent tasks, with an emphasis on the accuracy and robustness of supervision. Figure 4 summarizes the task synthesis pathways and the checks used to assess supervision accuracy and robustness.
我们通过五种互补的合成路径来扩展任务构建，旨在覆盖异构的编程语言、开发环境和任务范围。首先，在基于 GitHub 的合成中，我们收集拉取请求及其关联的问题，并应用启发式和模型辅助的过滤器来移除重复、格式错误或不可复现的候选。当拉取请求链接到提供了足够完整预期变更描述的问题时，我们使用该问题作为任务规范。否则，由 LLM 从参考补丁和周围仓库上下文重建问题风格的规范，同时明确指示其省略可能泄露解决方案的实现细节。其次，为了捕捉日常开发工作流，包括交互式“氛围编码”场景，我们收集组织内员工贡献的真实世界开发请求。这些请求涵盖功能实现、重构、调试和仓库维护等常见活动，智能体为这些活动生成并实现测试用例，将请求的行为操作化。第三，规范驱动的任务强调在详细的多约束要求下生成代码，使我们能够评估模型是否能够将复杂规范忠实转化为可执行实现。第四，在源代码驱动的合成中，我们使用 CodeMidas 从现有代码库中实现的功能派生任务。智能体识别候选功能，并将其可观察行为转化为任务规范和可执行环境，该路径使得跨多样仓库的可扩展任务构建成为可能，而无需问题、拉取请求或其他开发工件。对于长周期软件工程任务，智能体迭代地细化任务需求、扩展所需变更的范围、引入组件间的依赖关系，并构建测试以检验由此产生的多步行为。我们还包含并过滤来自公共来源和授权数据供应商的样本，以进一步丰富任务分布。总体而言，这些路径产生的任务涵盖广泛的编程语言、开发背景、实现复杂性和任务范围。
We scale task construction through five complementary synthesis pathways, designed to cover heterogeneous programming languages, development settings, and task horizons. First, in GitHub-based synthesis, we collect pull requests and their associated issues and apply heuristic and model-assisted filters to remove duplicate, malformed, or non-reproducible candidates. When a pull request is linked to an issue that provides a sufficiently complete description of the intended change, we use the issue as the task specification. Otherwise, an LLM reconstructs an issue-style specification from the reference patch and the surrounding repository context, while being explicitly instructed to omit implementation-specific details that could reveal the solution. Second, to capture everyday development workflows, including interactive “vibe coding” scenarios, we collect real-world development requests contributed by employees within our organization. These requests cover common activities such as feature implementation, refactoring, debugging, and repository maintenance, for which agents generate and implement test cases that operationalize the requested behavior. Third, specification-driven tasks emphasize code generation under detailed, multi-constraint requirements, allowing us to evaluate whether models can faithfully translate complex specifications into executable implementations. Fourth, in source-code-driven synthesis, we use CodeMidas to derive tasks from functionality implemented in existing codebases. Agents identify candidate functionality and translate its observable behavior into task specifications and executable environments, this pathway enables scalable task construction across diverse repositories without requiring issues, pull requests, or other development artifacts. For long-horizon software-engineering tasks, agents iteratively elaborate task requirements, expand the scope of the required changes, introduce dependencies across components, and construct tests that exercise the resulting multi-step behavior. We also include and filter samples from public sources and licensed data vendors to further enrich the task distribution. Collectively, these pathways produce tasks spanning a broad range of programming languages, development contexts, implementation complexities, and task horizons.
我们将每个任务规范的行为范围与其单元测试的行为范围对齐。目标是接受满足规范的实现，拒绝违反规范的实现，而不要求参考实现的偶然细节。我们审查任务是否覆盖不足和检查过于严格。特别是，强制执行规范中未出现要求的测试会被修改或移除。由于仅靠检查可能遗漏规范与测试之间的差异，我们进一步通过基于 rollout 的审计来评估它们的一致性。每个任务由编码智能体尝试四次。审计智能体在一个共享工作区中接收所有四次 rollout，该工作区包含问题陈述、测试、参考补丁以及每个 rollout 提交的补丁、测试输出和完整对话日志。审计员首先阐明正确解决方案的要求，并评估测试是否捕捉到这些要求。然后，它使用提交的补丁和对话日志，根据规范评估每个提交的解决方案，并将此评估与观察到的奖励进行比较。通过但被判定为不正确的解决方案被标记为潜在假阳性，表明验证不完整。失败但被判定为正确的解决方案被标记为潜在假阴性，表明测试过于严格或执行失败。这些分歧识别出需要进一步审查的任务，并提供具体证据表明预期行为与实施的验证之间可能存在不匹配。
We align the behavioral scope of each task specification with that of its unit tests. The goal is to accept implementations that satisfy the specification and reject those that violate it, without requiring incidental details of the reference implementation. We review tasks for insufficient coverage and overly restrictive checks. In particular, tests that enforce requirements absent from the specification are revised or removed. As inspection alone may miss discrepancies between specifications and tests, we further assess their alignment through rollout-based auditing. Each task is attempted four times by a coding agent. An auditing agent receives all four rollouts in a shared workspace containing the problem statement, tests, reference patch, and each rollout’s submitted patch, test output, and full conversation log. The auditor first articulates what a correct solution requires and assesses whether the tests capture those requirements. It then evaluates each submitted solution against the specification using its patch and conversation log, and compares this assessment with the observed reward. A passing solution judged incorrect is flagged as a potential false positive, suggesting incomplete verification. A failing solution judged correct is flagged as a potential false negative, suggesting overly restrictive tests or execution failures. These disagreements identify tasks requiring further review and provide concrete evidence of possible mismatches between the intended behavior and the implemented verification.
我们通过重复执行来检查奖励稳定性。对于有参考补丁的任务，我们确保在应用参考补丁之前，失败转通过（F2P）测试必须失败，通过转通过（P2P）测试必须通过；应用补丁后，两者都必须通过。我们要求这些结果在八次重跑中保持稳定，以筛查不稳定的测试和环境引起的奖励波动。我们还重用之前的 rollout 管道来收集智能体轨迹，以调查潜在的奖励黑客行为。提交的补丁和对话日志提供了智能体如何获得奖励的证据，有助于识别意外的捷径或对环境及评估过程的利用。这些检查补充了对解决方案正确性的评估，并为下文描述的更广泛的奖励黑客缓解程序提供信息。
We check reward stability through repeated execution. For those tasks with reference patch, we ensure fail-to-pass (F2P) tests must fail and pass-to-pass (P2P) tests must pass before applying the reference patch; after the patch, both must pass. We require these outcomes to remain stable across eight reruns, screening for flaky tests and environment-induced reward fluctuations. We also reuse the previous rollout pipeline to collect agent trajectories to investigate potential reward hacking. The submitted patches and conversation logs provide evidence of how agents obtained their rewards, helping identify unintended shortcuts or exploitation of the environment and evaluation process. These checks complement the assessment of solution correctness and inform the broader reward-hacking mitigation procedures described below.
总之，这些程序支持在一致的监督质量标准下构建多样化的编码任务语料库。规范-测试对齐针对奖励信号的保真度，重复执行检查其稳定性，基于 rollout 的审计探究其是否与解决方案正确性的独立评估一致。我们的目标是使这种组合具有可扩展性，以便 RL 训练能够受益于广泛的任务覆盖，同时鼓励智能体满足预期要求，而不是利用评估中的漏洞。
Together, these procedures support the construction of a diverse coding-task corpus under a consistent standard of supervision quality. Specification–test alignment targets the fidelity of the reward signal, repeated execution checks its stability, and rollout-based auditing probes whether it agrees with an independent assessment of solution correctness. Our objective is to make this combination scalable, so that RL training can benefit from broad task coverage while encouraging agents to satisfy the intended requirements rather than exploit gaps in evaluation.
我们将智能体能力扩展到跨多个领域的真实专业工作流中，这些工作流要求智能体分析异构文档、使用专用软件，并产出符合领域特定标准的交付物。我们合成支持此类工作流的环境，然后构建具有可验证结果的挑战性任务。图 5 总结了这一流程，包括环境构建、任务合成和评分标准（rubric）的迭代优化。
We extend agent capabilities to real-world professional workflows across diverse domains, whose workflows require agents to analyze heterogeneous documents, use specialized software, and produce deliverables that meet domain-specific standards. We synthesize environments that support such workflows, then construct challenging tasks with verifiable outcomes. Figure 5 summarizes this pipeline, including environment construction, task synthesis, and iterative rubric refinement.
通用智能体环境通常由一个工作区和一组软件工具组成，可通过 MCP、API、CLI 和 GUI 等接口访问。我们的设计在真实性与大规模强化学习（RL）的需求之间取得平衡：文件、软件行为和底层数据应反映专业实践，同时执行必须保持在本地，且环境必须易于重置。因此，我们收集真实文件，用于直接纳入或作为合成参考，并使用自动化的多智能体工作流构建本地软件模拟（mock），以复现所支持的操作、状态变化、输出格式和错误响应。对于已经无需外部服务即可运行的工具，我们直接集成。所有状态都存储在本地，每次 rollout 都在一个可恢复到固定初始状态的隔离沙箱中运行。这避免了网络波动和服务限制，同时支持可复现的执行。
A general-agent environment typically consists of a workspace and a set of software tools, accessible through interfaces such as MCP, APIs, CLIs, and GUIs. Our design balances realism with the requirements of large-scale RL: files, software behavior, and underlying data should reflect professional practice, while execution must remain local and environments must be easy to reset. We therefore collect real-world files for direct inclusion or as references for synthesis, and use an automated multi-agent workflow to build local software mocks that reproduce supported operations, state changes, output formats, and error responses. Tools that already operate without external services are integrated directly. All state is stored locally, and each rollout runs in an isolated sandbox that can be restored to a fixed initial state. This avoids network variability and service limits while supporting reproducible execution.
为了将这些组件组装成连贯的场景，规划智能体指定工作区结构、选择合适的工具，并规划文件和数据库内容。网络搜索使规划建立在真实世界信息之上，而文件之间以及文件与数据库记录之间的关系则支持后续的多跳推理和交叉验证。随后，多个智能体按照共享的规划和每个 mock 的数据规范并行生成文件并填充数据库。审查智能体检查单个产物和全局一致性，包括实体名称、数值核对、时间线和引用。在任务构建之前，迭代修复会解决不一致之处。工作区和工具配置因场景而异，因此每个环境都反映相关的工作实践，而不要求具备所有可用的文件类型或软件接口。
To assemble these components into coherent scenarios, a planning agent specifies the workspace structure, selects appropriate tools, and plans file and database contents. Web search grounds the plan in real-world information, while relationships across files and between files and database records support subsequent multi-hop reasoning and cross-validation. Multiple agents then generate files and populate databases in parallel, following the shared plan and each mock's data specification. A review agent checks individual artifacts and global consistency, including entity names, numerical reconciliation, timelines, and references. Iterative repair resolves inconsistencies before task construction. The workspace and tool configuration vary by scenario, so each environment reflects the relevant working practices without requiring every available file type or software interface.
我们调研常见的专业任务，并将它们组合、泛化为种子任务。智能体探索每个环境，并利用这些种子构建适合其内容和工具的任务。任务完成情况通过原子化的二元评分项进行评估：基于代码的检查验证确定性属性，例如数据库值和交付物格式，而基于 LLM 的检查则评估更开放的内容。对于基于 LLM 的评分项，同一模型重复判断之间以及不同评判模型之间的一致性有助于识别歧义。
We survey common professional tasks and combine and generalize them into seed tasks. An agent explores each environment and uses these seeds to construct tasks suited to its contents and tools. Task completion is evaluated using atomic, binary rubric items: code-based checks verify deterministic properties, such as database values and deliverable formats, while LLM-based checks assess more open-ended content. For LLM-based items, agreement across repeated judgments by the same model and across different judge models helps identify ambiguity.
仅凭一致性并不能证明评分标准捕捉到了任务完成情况。因此，我们收集来自不同能力水平模型的 rollout，并让审查智能体检查它们的轨迹、执行结果和评分判断。审查者会修改过于严格、会拒绝有效解决方案的标准，以及过于宽松、会接受不完整解决方案的标准。为了测试对奖励黑客（reward hacking）的抵抗力，我们加入负向检查，以检测对无关文件或数据库的非预期更改，并构建看似正确但实际并未解决任务的对抗性解决方案。在 RL 期间，一个自托管的 MiMo-V2.6-SFT 模型充当评分器，以支持稳定的评分。这一过程产生了数千个环境和多样化的可验证任务，我们将它们与其他任务结合用于 RL 训练。
Consistency alone does not establish that rubrics capture task completion. We therefore collect rollouts from models with different capability levels and have a review agent examine their trajectories, execution results, and rubric judgments. The reviewer revises overly restrictive criteria that reject valid solutions and overly permissive criteria that accept incomplete ones. To test resistance to reward hacking, we include negative checks for unintended changes to unrelated files or databases and construct adversarial solutions that appear correct without actually solving the task. During RL, a self-hosted MiMo-V2.6-SFT model serves as the grader to support stable scoring. This process yields thousands of environments and diverse, verifiable tasks, which we combine with other tasks for RL training.
为使智能体能够通过创造性表达与精确的视觉控制来塑造数字世界，我们精心设计了一套多样化的视觉任务，智能体在其中处理广泛类型的产物，例如网站、交互式应用、游戏、3D 场景、幻灯片、SVG、视频和 Figma 设计。我们将这些任务组织为两个互补的类别：开放式设计与高保真视觉复现。二者共同培养模型产出可靠实现、兼具出色美学质量与精确视觉控制的能力。
To empower agents to shape the digital world through creative expression and precise visual control, we curate a diverse suite of visual tasks in which agents work with a broad range of artifacts, such as websites, interactive applications, games, 3D scenes, slides, SVG, videos and Figma designs. We organize these tasks into two complementary categories: open-ended design and high-fidelity visual replication. Together, they develop the model's ability to produce reliable implementations with strong aesthetic quality and precise visual control.
开放式设计旨在产出满足用户意图、具有美学吸引力的视觉产物。有效创意方案的多样性使得仅靠固定规则难以刻画美学质量。因此，我们结合逐点评分与组内评分，以建立一致的质量标准，同时对美学质量的相对提升给予奖励。具体而言，我们首先迭代细化针对运行时正确性、指令遵循、布局完整性和基本美学质量的逐点评分细则。待这些细则稳定后，我们引入组内评分，对同一查询下每个 rollout 组内渲染出的产物进行联合比较，识别出明显更强与更弱的候选，并据此分配奖励。
Open-ended design seeks to produce aesthetically compelling visual artifacts that satisfy user intent. The diversity of valid creative solutions makes aesthetic quality difficult to capture through fixed rules alone. We therefore combine pointwise and groupwise grading to establish consistent quality standards while rewarding relative improvements in aesthetic quality. Specifically, we first iteratively refine pointwise rubrics for runtime correctness, instruction adherence, layout integrity, and basic aesthetic quality. Once these rubrics stabilize, we introduce groupwise grading that jointly compares the rendered artifacts within each rollout group for the same query, identifying clearly stronger and weaker candidates and assigning rewards accordingly.
高保真视觉复现旨在忠实复现指定的视觉目标。与开放式设计相比，明确的参考对象使得对视觉保真度的评估更为直接。我们主要通过基于规则的相似度指标对这些任务进行评分，例如渲染产物与其参考之间的像素级相似度，并辅以基于 LLM 的评判以进行整体视觉评估。这些信号共同支撑了强化学习在视觉复现任务上的有效 Scaling（规模扩张）。
High-fidelity visual replication aims to faithfully reproduce a specified visual target. Compared to open-ended design, the explicit reference enables more direct evaluation of visual fidelity. We grade these tasks primarily through rule-based similarity metrics, such as pixel-level similarity between rendered artifacts and their references, supplemented by LLM-based judging for holistic visual assessment. Together, these signals support effective scaling of reinforcement learning on visual replication tasks.
我们使用强化学习在真实世界的漏洞复现任务上训练网络安全智能体：给定一个项目和目标漏洞，智能体必须构造一个能触发该漏洞的输入。这一基础攻击性安全技能是漏洞利用开发、权限提升和 CTF 挑战的基础。它也适合大规模强化学习：OSS-Fuzz 提供了数万个手动确认的实例，这一数量是其他安全任务无法比拟的。难点不在于触发崩溃，因为复杂的 C/C++项目会暴露数十条可达的崩溃路径，而在于触发所描述的特定漏洞。因此，验证必须将目标漏洞与无关崩溃区分开，并且由于它同时作为强化学习的奖励，它必须准确、确定且廉价。
We train cyber agents with RL on real-world vulnerability reproduction: given a project and a target bug, the agent must construct an input that triggers it. This foundational offensive-security skill underlies exploit development, privilege escalation, and CTF challenges. It is also well suited to RL at scale: OSS-Fuzz supplies tens of thousands of manually confirmed instances, a volume unmatched by other security tasks. The difficulty is not triggering a crash, since complex C/C++ projects expose dozens of reachable crash paths, but triggering the specific vulnerability described. Verification must therefore separate the intended bug from unrelated crashes, and because it doubles as the RL reward, it must be accurate, deterministic, and cheap.
现有的判定标准存在不足。CyberGym 中使用的修复二进制差分测试接受一个能使存在漏洞的二进制崩溃但不能使修补后的二进制崩溃的 PoC；不完整的补丁会拒绝正确的 PoC，而两个提交之间的无关更改会因与漏洞无关的原因翻转判定结果。任一错误都会破坏训练梯度。基于 LLM 的评判对同一个 PoC 在不同运行中返回不同的判定，无法作为稳定的奖励。相反，我们从真实的标准检查器（ASan/MSan/UBSan）报告中提取两个属性：漏洞类型（例如，堆缓冲区溢出）和崩溃位置（最顶层的项目级堆栈帧）。当且仅当 PoC 的崩溃在基于规则的字符串匹配下同时匹配这两个属性时，才被接受，这种匹配是确定性的、可复现的且计算上微不足道。任务描述源自同一报告，说明崩溃必须发生的准确类型和函数，因此描述和验证共享单一事实来源；相比之下，CyberGym 的 LLM 生成的描述通常过于宽泛（例如，“libxml2 中的缓冲区溢出”）或完全错误。
Existing oracles fall short. Fix-binary differential testing, as used in CyberGym, accepts a PoC that crashes the vulnerable binary but not the patched one; an incomplete patch rejects correct PoCs, and unrelated changes between the two commits flip verdicts for reasons unconnected to the bug. Either error corrupts the training gradient. LLM-based judging returns different verdicts for the same PoC across runs and cannot serve as a stable reward. We instead extract two attributes from the ground-truth sanitizer (ASan/MSan/UBSan) report: the vulnerability type (e.g., heap-buffer-overflow) and the crash location (the topmost project-level stack frame). A PoC is accepted iff its crash matches both under rule-based string matching, which is deterministic, reproducible, and computationally trivial. The task description is derived from the same report, stating the exact type and function in which the crash must occur, so description and verification share a single source of truth; CyberGym’s LLM-generated descriptions, by contrast, are often too broad (e.g., “buffer overflow in libxml2”) or simply wrong.
对于每个漏洞，我们检出报告提交处的源代码，构建模糊测试框架，并为智能体提供完整的运行时环境：完整源代码加上编译后的框架二进制文件。CyberGym 不提供二进制文件，限制智能体仅进行源代码分析；提供二进制文件反映了真实的漏洞分析，研究人员在调试器下运行目标，检查内存，并根据运行时观察构造变异。
For each vulnerability we check out the source at the reported commit, build the fuzzing harness, and give the agent the complete runtime environment: full source plus the compiled harness binary. CyberGym withholds the binary, restricting agents to source-only analysis; providing it mirrors real vulnerability analysis, where a researcher runs the target under a debugger, inspects memory, and crafts mutations from runtime observations.
智能体执行框架在模块设计上各不相同，且是为满足不同应用需求而选择或构建的。因此，模型必须适应多样的交互机制，包括训练期间未曾见过的执行框架的交互机制，这使得跨执行框架泛化成为一项重要能力。
Agent harnesses differ in module design and are selected or built to meet different application needs. Models must therefore adapt to diverse interaction mechanisms, including those of harnesses unseen during training, making cross-harness generalization an important capability.
仅在单一执行框架内训练，可能使任务求解策略与特定执行框架的实现相耦合，从而限制迁移。因此，我们将执行框架的多样性视为与任务多样性和环境多样性并列的额外训练维度。一种自然的做法是在 MiMo Code 和 Codex 等多个生产级执行框架上进行训练。然而，在实践中这并不适合强化学习。首先，生产级执行框架用工程化防护措施和多步工作流包裹智能体循环，并通过大量约束与指令提示引导模型。这些额外内容落在任务完成奖励信号之外，使信用分配不可靠，并导致未被奖励衡量的要求被直接忽略。其次，它们的模块紧密耦合而非可独立配置，因此无法单独改变某一交互机制，也无法通过受控重组来组合多样性。
Training exclusively within a single harness may couple task-solving strategies to harness-specific implementations and limit transfer. We therefore treat harness diversity as an additional training dimension alongside diversity in tasks and environments. A natural approach is to train on several production harnesses such as MiMo Code and Codex. However, this is a poor fit for RL in practice. First, production harnesses wrap the agent loop with engineering safeguards and multi-step workflows, and steer the model with numerous constraint and instruction prompts. These extras fall outside the task-completion reward signal, making credit assignment unreliable and leaving reward-unmeasured requirements to simply be ignored. Second, their modules are tightly coupled rather than independently configurable, so one cannot vary a single interaction mechanism in isolation or assemble diversity by controlled recombination.
为此，我们提出使用精心设计的迷你执行框架进行多执行框架训练。所有迷你执行框架都从同一个最小智能体循环出发，该循环已包含完成任务所需的组件——系统提示、工具和上下文管理。这些模块保持最小化且解耦，因此实现可以自由重组，执行框架的扩展也保持可控、安全且简洁。随后，我们为 Code、General、Visual 和 Cyber 推导出多样且适配任务的迷你执行框架配置。由此得到的训练分布既多样又可控，支持对单个执行框架机制进行分析，并鼓励形成可迁移的任务求解策略。
To this end, we propose multi-harness training with carefully designed mini-harnesses. All mini-harnesses start from the same minimal agent loop, which already includes the components needed for task completion—system prompt, tools, and context management. These modules stay minimal and decoupled, so implementations can be freely recombined and harness expansion remains controllable, safe, and clean. We then derive diverse, task-adapted mini-harness configurations for Code, General, Visual, and Cyber. The resulting training distribution is diverse yet controllable, supporting analysis of individual harness mechanisms and encouraging transferable task-solving strategies.
强化学习依赖奖励来衡量任务是否成功，但智能体有时会通过利用环境或评估过程来获取高奖励。这种行为被称为奖励黑客行为（reward hacking），它可能在训练过程中被强化，从而在不提升任务表现的情况下提高分数。对于仓库修复（repository-repair）这类常见的编程智能体任务，一种反复出现的失败模式是解法泄漏（solution leakage）：智能体获取了超出预期任务上下文的已发布修复，并用它来构造补丁。表 2 在五个仓库上，将每个任务请求与智能体所声明的意图及后续动作对应起来。此类行为可以满足基于测试的奖励，却无法证明智能体是从缺陷报告和指定的代码检出（checkout）中推导出修复的。为此，我们提出了相应的缓解措施，将中期训练对齐数据与 RL 环境准备、对抗性筛查以及贯穿训练全程的审计相结合。
RL relies on rewards to measure task success, but agents can sometimes earn high rewards by exploiting the environment or the evaluation process. This behavior, known as reward hacking, can be reinforced during training, increasing scores without improving task performance. For common coding agent tasks like repository-repair, a recurring failure mode is solution leakage: agents obtain a published fix beyond the intended task context and use it to construct a patch. Table 2 links each task request to the agent's stated intent and subsequent action across five repositories. Such behavior can satisfy the test-based reward without demonstrating that the agent derived the repair from the bug report and the assigned checkout. Following this, we introduce our mitigation combining mid-training alignment data with RL environment preparation, adversarial screening, and auditing throughout training.
在早期实验中，我们观察到 MiMo 存在奖励黑客行为的倾向。为缓解这一行为，我们从这些案例中合成了一组训练样本，并将其纳入中期训练。对于每个案例，MiMo 会反思错误的推理，修正相关轮次，并继续执行以任务规范为依据的动作。修正后的推理保留了原始错误以便识别，同时使纠正过程显式化。我们发现，加入这些样本提升了模型的对齐程度。
In early experiments, we observed a tendency toward reward hacking in MiMo. To mitigate this behavior, we synthesized a set of training examples from these cases and included them in mid-training. For each case, MiMo reflects on the faulty reasoning, revises the relevant turn, and continues with actions grounded in the task specification. The revised reasoning keeps the original error recognizable and makes the correction explicit. We found that adding these examples improved the model's alignment.
在编程任务中，智能体可能通过从环境中恢复泄漏的解法，或通过网络检索已有解法来通过测试。在环境构建期间应用参考补丁来构建项目，可能会留下暴露解法的产物。因此，我们移除构建日志、验证器输出、残留补丁，以及可能暴露解法的项目生成二进制文件或字节码。我们还清理可能包含解法信息的缓存，包括仓库之外的缓存，同时保留离线重建所需的第三方依赖。对于每个任务，我们保留截至并包含基础提交（base commit）的 Git 历史，移除之后的提交及其相关引用。我们实施容器级网络隔离，以限制对上游修复以及可能包含解法的其他包版本的访问。这些防护措施还配有明确指令，禁止检索已有解法或绕过所要求的实现。
In coding tasks, agents may pass tests by recovering leaked solutions from the environment or retrieving existing solutions over the network. Building a project with the reference patch applied during environment construction can leave behind artifacts that reveal the solution. We therefore remove build logs, verifier outputs, residual patches, and project-generated binaries or bytecode that could expose it. We also clean caches that may contain solution information, including those outside the repository, while preserving third-party dependencies needed for offline rebuilding. For every task, we retain Git history up to and including the base commit, removing later commits and their associated references. We enforce container-level network isolation to restrict access to upstream fixes and alternative package versions that may contain the solution. These safeguards are accompanied by explicit instructions against retrieving existing solutions or bypassing the required implementation.
随后，一个专门的 hack 智能体（hack agent）会探测已准备好的环境，寻找残留泄漏和可利用的弱点。我们用早期实验中的示例来引导其搜索，包括从缓存产物或目标项目的预装副本中恢复解法。该智能体在检查这些已知路径的同时，也在搜索新的路径。它发现了许多我们在训练期间未曾观察到、且现有清理流程未能覆盖的利用路径。我们利用这些发现来改进清理和访问限制，然后在更新后的环境中重新运行 hack 智能体。这些后续检查反复暴露出更多弱点，需要额外多轮清理和测试。我们持续这一过程，直到 hack 智能体无法在任何环境中找到成功的利用路径为止。
A dedicated hack agent then probes the prepared environments for remaining leaks and exploitable weaknesses. We guide its search with examples from early experiments, including recovering solutions from cached artifacts or preinstalled copies of the target project. The agent checks these known routes while searching for new ones. It uncovered many exploit paths that we had not observed during training and that our existing cleanup procedures did not cover. We use these findings to refine cleanup and access restrictions, then rerun the hack agent on the updated environments. These subsequent checks repeatedly exposed further weaknesses, requiring additional rounds of cleanup and testing. We continued this process until the hack agent could no longer find a successful exploit in any of the environments.
我们在整个训练过程中定期离线审计智能体的轨迹，以检查奖励黑客行为。随着策略的演化，它可能发现对抗性筛查中未能发现的捷径。我们利用这些观察来识别环境中的弱点，并指导进一步的清理和访问限制。除这些离线审计外，分组式智能体评分器（groupwise agentic grader，第 4.3.2 节）会在重新计算组统计量和优势之前，将已确认的 hack 轨迹的有效奖励设为零。在此修正下，对于 MiMo-V2.6-Flash 和 MiMo-V2.6-Pro，整个训练过程中记录的已确认 hack 占比始终低于 \(2\)%（图 6）。
We regularly audit agent trajectories offline for reward hacking throughout training. As the policy evolves, it may discover shortcuts that were not found during adversarial screening. We use these observations to identify weaknesses in the environments and guide further cleanup and access restrictions. Alongside these offline audits, the groupwise agentic grader (Section 4.3.2) sets the effective reward of confirmed hacking trajectories to zero before recomputing group statistics and advantages. With this correction in place, the logged confirmed-hack share remains below \(2\)% throughout the whole training process for both MiMo-V2.6-Flash and MiMo-V2.6-Pro (Figure 6).
对于代码智能体任务，二元测试奖励提供了可扩展的正确性信号，但无法区分通过解之间的实现质量或问题求解行为。我们在这些任务的不同子集上使用两种互补的方法。分组式奖励合成（Groupwise Reward Synthesis，GRS）用于高通过率任务的一个子集。它比较多个离线 rollout 以构建任务特定的评分细则（rubrics），这些细则在训练期间被复用以对单个 rollout 打分，并将其质量分数与测试奖励相结合。对于其余所有代码智能体任务，我们主要依赖分组式优势重分配（Groupwise Advantage Redistribution，GAR）。其在线智能体式评分器联合检查每个混合结果组内的成功与失败轨迹，对通过解进行排序，并将正优势重新分配给质量更高的通过轨迹。图 7 总结了这两种工作流程。
For code agent tasks, binary test rewards provide a scalable correctness signal but do not distinguish implementation quality or problem-solving behavior among passing solutions. We use two complementary methods on distinct subsets of these tasks. Groupwise Reward Synthesis (GRS) is used for a subset of high-pass-rate tasks. It compares multiple offline rollouts to construct task-specific rubrics, which are reused during training to score individual rollouts and combine their quality scores with test rewards. For all the remaining code agent tasks, we rely primarily on Groupwise Advantage Redistribution (GAR). Its online agentic grader jointly examines successful and failed trajectories within each mixed-outcome group, ranks passing solutions, and redistributes positive advantage toward higher-quality passing trajectories. Figure 7 summarizes both workflows.
对于每个选定的任务，我们收集多条离线 rollout，并让一个智能体结合任务说明与代码仓库对它们进行研究。比较这些尝试可以揭示不同的解法、反复出现的错误，以及可能分散在若干条轨迹中的有用行为。智能体将这一分析转化为两组标准：**解法评分标准**（solution rubrics），用于评估最终实现；以及**行为评分标准**（behavior rubrics），用于评估智能体如何着手并验证其工作。
For each selected task, we collect multiple offline rollouts and ask an agent to study them together with the task specification and repository. Comparing these attempts exposes different solution approaches, recurring mistakes, and useful behaviors that may be distributed across several trajectories. The agent turns this analysis into two sets of criteria: solution rubrics, which assess the resulting implementation, and behavior rubrics, which assess how the agent approaches and verifies its work.
解法评分标准描述一个良好实现中与任务相关的属性，包括是否满足需求、是否恰当处理边界情况，以及与周边代码库保持一致的改动。行为评分标准描述支持可靠问题解决的可观察实践，例如收集相关证据、检查代码改动的效果。我们寻求能够区分质量上实质性差异、同时允许对任务采用不同有效方法的标准。
Solution rubrics describe task-relevant properties of a good implementation, including satisfaction of the requirements, appropriate handling of edge cases, and changes consistent with the surrounding codebase. Behavior rubrics describe observable practices that support reliable problem solving, such as gathering relevant evidence and checking the effects of code changes. We seek criteria that distinguish meaningful differences in quality while allowing different valid approaches to the task.
采样的 rollout 为评分标准的构建提供依据，但标准本身以任务为根基。一个有用的行为可能只出现在一次尝试中，而任务所支持的一项需求也可能在所有已观察到的解法中都不存在。因此，智能体能够识别出超出采样 rollout 所展示的改进。与此同时，某个成功解法所做的选择并不会自动成为所有其他解法的要求。
The sampled rollouts inform rubric construction, but the criteria are grounded in the task itself. A useful behavior may appear in only one attempt, and a requirement supported by the task may be absent from every observed solution. The agent can therefore identify improvements beyond those demonstrated in the sampled rollouts. At the same time, a choice made by one successful solution does not automatically become a requirement for all others.
所得的评分标准被复用于对后续训练 rollout 逐一评估。一个评分智能体进入每条 rollout 的执行环境，并依据任务特定的评分标准对其进行评估，以最终代码、执行结果和轨迹作为证据。它为实现质量给出一个**解法分数** \(S^{\mathrm{sol}}\)，并为智能体如何着手与验证其工作给出一个**行为分数** \(S^{\mathrm{beh}}\)。
The resulting rubrics are reused to evaluate subsequent training rollouts individually. A grader agent enters each rollout's execution environment and assesses it against the task-specific rubrics, using the resulting code, execution results, and trajectory as evidence. It assigns a solution score \(S^{\mathrm{sol}}\) for the quality of the implementation and a behavior score \(S^{\mathrm{beh}}\) for how the agent approached and verified its work.
对于轨迹 \(i\)，令 \(R_{i}^{\mathrm{test}}\) 表示原始的二元测试奖励。我们通过将该测试奖励与两个评分标准分数相乘，来合成最终训练奖励：
For trajectory \(i\), let \(R_{i}^{\mathrm{test}}\) denote the original binary test reward. We synthesize the final training reward by multiplying this test reward by the two rubric scores:
这种乘法形式使评分细则监督与测试结果保持关联。失败的轨迹奖励保持为零，而通过的轨迹则依据实现质量和问题求解行为被进一步区分。即使一个组内的所有 rollout 都通过测试，两个评分细则分数乘积的差异仍能提供学习信号。因此，离线任务分析成为一种可复用的监督来源，使训练能够捕捉二元测试奖励未能表达的质量差异。
This multiplicative form keeps rubric supervision tied to test outcomes. Failed trajectories retain zero reward, while passing trajectories are further distinguished by implementation quality and problem-solving behavior. Even when every rollout in a group passes the tests, differences in the product of the two rubric scores can still provide a learning signal. The offline task analysis thus becomes a reusable source of supervision, allowing training to capture quality differences that binary test rewards leave unexpressed.
我们将在线分组优势重分配应用于剩余的代码智能体任务。对于每个混合结果的 rollout 组，我们将所有轨迹放入一个共享工作区，其中包含任务规范、代码仓库、提交的补丁和测试输出。一个经过 SFT 训练的智能体式评分器会联合检查组内的所有轨迹，对比成功和失败的尝试，并从五个维度比较通过的补丁：解决方案的适用性、实现该方案的精确性（无遗漏或不必要的回退）、相对于必要更改的最小性、避免任务之外的意外影响，以及与代码库约定一致的手艺。这些评估指导分组排名，当差异不确定时则并列。评分器可以检查仓库代码并运行针对性测试，以更好地理解任务并验证候选解决方案。当证据确认依赖于外部或泄露的答案时，我们将轨迹的有效奖励重置为零，并在重新计算组统计量之前将其视为失败。
We apply online groupwise advantage redistribution to the remaining code agent tasks. For each mixed-outcome rollout group, we place all trajectories in a shared workspace containing the task specification, repository, submitted patches, and test outputs. An SFT-trained agentic grader jointly examines all trajectories within the group, contrasting successful and failed attempts and comparing passing patches along five dimensions: the suitability of the solution approach, precision in implementing that approach without omissions or unnecessary fallbacks, minimality relative to the necessary changes, avoidance of unintended effects outside the task, and craftsmanship consistent with codebase conventions. These assessments guide the groupwise ranking, with ties when differences are inconclusive. The grader can inspect repository code and run targeted tests to better understand the task and verify candidate solutions. When evidence confirms dependence on an external or leaked answer, we reset the trajectory’s effective reward to zero and treat it as a failure before recomputing group statistics.
我们使用剩余的质量排名来重分配序列级优势。对于轨迹 \(i\)，令 \(R_{i}\) 为 hack 修正后的有效二元奖励，\(\bar{R}\) 为其组均值，\(A_{i}=R_{i}-\bar{R}\) 为其序列级优势，\(\mathcal{P}=\{i:R_{i}=1\}\)。对于非空的 \(\mathcal{P}\)，质量因子 \(f_{i}\in(0,1]\) 首先降低低质量通过轨迹的权重，然后一个公共因子将移除的正优势质量重新分配给通过的轨迹。未截断的更新为
We use the remaining quality rankings to redistribute sequence-level advantages. For trajectory \(i\), let \(R_{i}\) be the effective binary reward after hack correction, \(\bar{R}\) its group mean, \(A_{i}=R_{i}-\bar{R}\) its sequence-level advantage, and \(\mathcal{P}=\{i:R_{i}=1\}\). For nonempty \(\mathcal{P}\), quality factors \(f_{i}\in(0,1]\) first downweight lower-quality passes, after which a common factor redistributes the removed positive advantage mass among passing trajectories. The uncapped update is
这种未截断的更新保持了 \(\sum_{i\in\mathcal{P}}A_{i}^{\prime}=\sum_{i\in\mathcal{P}}A_{i}\) 以及质量诱导的相对权重，而不改变失败的轨迹。实际上，它将正优势质量从低质量的成功轨迹重新分配给高质量的成功轨迹。简单地降低正优势的权重会使负优势保持不变；重新归一化恢复了它们的平衡，作为防止熵过度增长的保障。在实践中，我们对公共重缩放因子进行截断，以防止正优势的过度放大。然后，我们从通过和失败轨迹的优势中减去组均值，得到组均值为零的最终序列优势 \(A_{i}^{\mathrm{new}}\)。我们将这个序列级优势广播到轨迹中的所有响应 token。评分异步运行，不可用的评分器输出回退到原始优势。
This uncapped update preserves \(\sum_{i\in\mathcal{P}}A_{i}^{\prime}=\sum_{i\in\mathcal{P}}A_{i}\) and the quality-induced relative weights without altering failed trajectories. In effect, it redistributes positive advantage mass from lower-quality to higher-quality successful trajectories. Simply downweighting positive advantages leaves negative advantages unchanged; renormalization restores their balance as a safeguard against excessive entropy growth. In practice, we cap the common rescaling factor to prevent excessive amplification of positive advantages. We then subtract the group mean from the advantages of both passing and failed trajectories, yielding final sequence advantages \(A_{i}^{\mathrm{new}}\) with zero group mean. We broadcast this sequence-level advantage to all response tokens in the trajectory. Grading runs asynchronously, with unusable grader outputs falling back to the original advantages.
为了评估在线分组优势重分配对训练动态的影响，我们比较了 MiMo-V2.6-Flash 在仅代码 RL 运行中有无该方法的差异，使用训练批量大小为 128 和 token 均值损失聚合（图 8）。没有在线评分时，轮次和总 token 长度快速增长，导致更多轨迹达到长度限制，难以持续提高通过率。有在线评分时，通过率的提升持续到第 52 步，而轮次数保持大致稳定，token 长度逐渐增长。这些趋势表明，在线评分支持在更稳定的训练机制下持续进行策略改进。
To assess the effect of online groupwise advantage redistribution on training dynamics, we compare code-only RL runs of MiMo-V2.6-Flash with and without it, using a training batch size of 128 and token-mean loss aggregation (Figure 8). Without online grading, turns and total token length grow rapidly, causing more trajectories to hit the length limit, making it difficult to sustain improvements in pass rate. With online grading, pass-rate gains are sustained through step 52, while turn counts remain roughly stable and token length grows gradually. These trends suggest that online grading supports continued policy improvement under a more stable training regime.
独立的面向维护者的审计发现，在提高测试通过率的压力下，未经在线评分的策略越来越多地采用不良行为，如推测性兼容分支、广泛导出、异常吞没、放宽验证以及针对评估的配置更改。这些变通方法旨在增加通过测试的可能性，但可能超出任务指令的范围，不必要地扩展 API，掩盖失败，并使代码更难维护。相比之下，经过在线评分的策略倾向于产生更小、更精确的补丁，这些补丁保持在请求的范围内，并且更易于维护。
Separate maintainer-oriented audits found that, under pressure to improve test pass rates, policies trained without online grading increasingly adopted undesirable behaviors such as speculative compatibility branches, broad exports, exception swallowing, relaxed validation, and evaluation-specific configuration changes. These workarounds aim to increase the likelihood of passing test but can exceed the scope of the task instructions, expand APIs unnecessarily, obscure failures, and make the code harder to maintain. In contrast, policies trained with online grading tended to produce smaller, more precise patches that remained within the requested scope and were easier to maintain.
为提高强化学习训练的稳定性并鼓励高效、可靠的行为，我们引入两种互补机制。组相对长度惩罚抑制 token 的过度增长，而片段级行为惩罚处理格式违规与工具调用错误。批级优势重平衡改进信用分配，同时限制可能驱动熵失控增长的过度负向优化压力。
To improve RL training stability and encourage efficient, reliable behavior, we introduce two complementary mechanisms. A group-relative length penalty curbs excessive token growth, while segment-level behavioral penalties address format violations and tool call errors. Batch-level advantage rebalancing improves credit assignment while limiting excess negative optimization pressure that can drive uncontrolled entropy growth.
我们发现，组相对长度惩罚可改善泛化，并在强化学习训练期间抑制生成 token 的快速增长。对于每个提示 \(q\) 及其 \(G\) 条采样轨迹，令 \(R_{i}\) 为原始结果奖励，\(\mathcal{P}_{q}\) 为成功轨迹的索引集合，\(\ell_{i}=|o_{i}|\) 为生成的 token 数。令 \(A\in[0,1]\) 为最小组通过率，\(B\in(0,100)\) 为分位数参数。对于满足 \(|\mathcal{P}_{q}|/G>A\) 的组，我们计算参考长度 \(\ell_{q}^{\star}=\operatorname{Quantile}_{B/100}\{\ell_{j}:j\in\mathcal{P}_{q}\}\) 并得到调整后的奖励
We find that a group-relative length penalty improves generalization and curbs rapid growth in generated tokens during RL training. For each prompt \(q\) with \(G\) sampled rollouts, let \(R_{i}\) be the original outcome reward, \(\mathcal{P}_{q}\) the indices of successful rollouts, and \(\ell_{i}=|o_{i}|\) the generated-token count. Let \(A\in[0,1]\) be the minimum group pass rate and \(B\in(0,100)\) the percentile parameter. For groups with \(|\mathcal{P}_{q}|/G>A\), we compute the reference length \(\ell_{q}^{\star}=\operatorname{Quantile}_{B/100}\{\ell_{j}:j\in\mathcal{P}_{q}\}\) and obtain the adjusted reward
其中 \(X\geq 0\) 为最大奖励扣减量，\(\delta\geq 0\) 为超过 \(\ell_{q}^{\star}\) 所容忍的相对超出量，\(s>\delta\) 为惩罚饱和时的超出量，\(\gamma\geq 1\) 为斜坡指数；\(\delta=0\) 会对长于参考长度的成功轨迹施加惩罚。指示函数 \(\mathbf{1}[\cdot]\) 将扣减限制在成功轨迹上，且 \(\operatorname{clip}(x,0,1)=\min(1,\max(0,x))\)。其他组保留其原始奖励。式 (1) 中的结果优势 \(A_{i}\) 由调整后的奖励 \(\widetilde{R}_{i}\) 计算得到；阈值 \(A\) 是单独的超参数。这利用针对每个提示自适应的参考长度来鼓励简洁的成功解，而通过率门控则为困难提示保留了探索空间。
Here \(X\geq 0\) is the maximum reward deduction, \(\delta\geq 0\) the tolerated relative excess above \(\ell_{q}^{\star}\), \(s>\delta\) the excess at which the penalty saturates, and \(\gamma\geq 1\) the ramp exponent; \(\delta=0\) penalizes successful rollouts longer than the reference length. The indicator \(\mathbf{1}[\cdot]\) restricts the deduction to successful rollouts, and \(\operatorname{clip}(x,0,1)=\min(1,\max(0,x))\). Other groups retain their original rewards. The outcome advantage \(A_{i}\) in Eq. (1) is computed from the adjusted rewards \(\widetilde{R}_{i}\); the threshold \(A\) is a separate hyperparameter. This encourages concise successful solutions using a reference adapted to each prompt, while the pass-rate gate preserves room for exploration on difficult prompts.
结果奖励可能强化错误的中间行为，这促使我们针对格式违规与工具调用错误（如标记格式错误、工具名称无效或参数格式错误）制定片段级规则。令 \(h_{i,t}=1\) 标记被标记的 token，否则为 \(0\)。在整个训练批次上，\(H_{\pm}\) 与 \(C_{\pm}\) 分别收集损失掩码 \(M_{i,t}=1\) 下被标记与未被标记的 token 索引 \((i,t)\)；\(\pm\) 表示所属轨迹结果优势 \(A_{i}\) 的符号，下文所有求和均按 token 计数。
Outcome rewards can reinforce faulty intermediate behavior, motivating segment-level rules for format violations and tool call errors such as malformed markup, invalid tool names, or malformed arguments. Let \(h_{i,t}=1\) mark flagged tokens and \(0\) otherwise. Across the full training batch, \(H_{\pm}\) and \(C_{\pm}\) collect flagged and unflagged token indices \((i,t)\) with loss mask \(M_{i,t}=1\), respectively; \(\pm\) denotes the sign of the owning trajectory’s outcome advantage \(A_{i}\), and all sums below count tokens.
其中 \(\kappa>1\) 放大被标记 token 上负优势的幅度；\(\alpha\) 将未被标记的正向 token 向上缩放，而 \(\beta\) 将未被标记的负向 token 向零缩放。超参数 \(\alpha_{\max}\geq 1\) 与 \(0<\beta_{\min}\leq 1\) 分别对正向放大设上限并对负向衰减设下界。调整后的 token 优势 \(\widetilde{A}_{i,t}\) 替换式 (1) 中的 \(A_{i}\)。这在正向轨迹中屏蔽被标记的 token，并在负向轨迹中对其施加更强惩罚。被移除的正向优势被重新分配给未被标记的正向 token，而增加的负向幅度则通过减小对未被标记的负向 token 的惩罚来抵消：未检测到错误的行为会根据轨迹的符号获得更多强化或更少惩罚。当两个缩放均未被裁剪时，每个符号的总优势质量守恒，从而限制可能驱动熵失控增长的过度负向压力。若某分母为零，则其缩放设为 1；在此情形或发生裁剪时，不保证守恒。
Here \(\kappa>1\) multiplies the magnitude of negative advantage on flagged tokens; \(\alpha\) scales unflagged positive tokens upward, while \(\beta\) scales unflagged negative tokens toward zero. The hyperparameters \(\alpha_{\max}\geq 1\) and \(0<\beta_{\min}\leq 1\) cap positive amplification and bound negative attenuation, respectively. The adjusted token advantage \(\widetilde{A}_{i,t}\) replaces \(A_{i}\) in Eq.(1). This masks flagged tokens in positive trajectories and penalizes them more strongly in negative trajectories. Removed positive advantage is redistributed to unflagged positive tokens, while added negative magnitude is offset by reducing penalties on unflagged negative tokens: behavior without detected errors receives more reinforcement or less punishment, depending on the trajectory’s sign. Each sign’s total advantage mass is conserved when neither scale is clipped, limiting excess negative pressure that can drive uncontrolled entropy growth. If a denominator is zero, its scale is set to one; conservation is not guaranteed in this case or when clipping occurs.
本节介绍大规模强化学习（RL）训练运行，包括训练设置（§5.1）、评估设置（§5.2）、RL 性能（§5.3）、专家负载稳定性（§5.4），并对失败情况进行分析（§5.5）。运行日志可在 https://mimo.xiaomi.com/rl/mimo-v26 查看。
In this section, we describe the scaled RL training run, including the training setup (§5.1), evaluation settings (§5.2), RL performance (§5.3), expert load stability (§5.4), and an analysis of the failures (§5.5). The running log can be found at https://mimo.xiaomi.com/rl/mimo-v26.
我们的强化学习任务涵盖多个领域，包括智能体式与竞赛编程（68%）、通用工具使用（12%）、美学设计（13%）、上下文遵循（3%）以及网络安全（4%）。我们采样一个包含 1568 条提示的批次，每条提示进行 16 次 rollout，总计形成 25K 条轨迹的全局训练批次。训练使用 GRPO，并采用异步部分 rollout，陈旧度为 4。
Our RL tasks span various domains, including agentic and competitive coding (68%), general tool use (12%), aesthetic design (13%), context following (3%), and cyber security (4%). We sample a prompt batch of 1568 with 16 rollouts per prompt, totaling a global train batch of 25K trajectories. Training uses GRPO with asynchronous partial rollouts at a staleness of 4.
我们使用 Muown 优化器，学习率为 \(3\times 10^{-6}\)，不使用权重衰减或学习率预热，梯度裁剪阈值为 \(1.0\)。对于 Muon 组件，我们使用动量系数 \(0.95\) 并启用 Nesterov 动量，每次更新执行 \(10\) 次 Newton–Schulz 迭代，并施加额外的更新缩放因子 \(0.5\)。对于 Adam 组件，我们设置 \(\beta_{1}=\beta_{2}=0.95\) 和 \(\epsilon=10^{-8}\)。为稳定 MXFP4 训练，我们通过从 SFT 检查点继承 FP32 主权重和 Muown 的行状态来初始化强化学习。在强化学习训练期间，我们冻结路由器以维持稳定的专家负载。
We use the Muown optimizer with a learning rate of \(3\times 10^{-6}\), no weight decay or learning rate warmup, and a gradient clipping threshold of \(1.0\). For the Muon component, we use a momentum coefficient of \(0.95\) with Nesterov momentum enabled, perform \(10\) Newton–Schulz iterations per update, and apply an additional update scaling factor of \(0.5\). For the Adam component, we set \(\beta_{1}=\beta_{2}=0.95\) and \(\epsilon=10^{-8}\). To stabilize MXFP4 training, we initialize RL by carrying over the FP32 master weights and Muown's row state from the SFT checkpoint. During RL training, we freeze the router to maintain stable expert loads.
我们采用式 (1) 中的组相对策略优化（GRPO），优势计算详见 §4.3。对于损失聚合，我们使用提示均值聚合，即在提示级别对替代目标求平均，而非对所有响应 token 求平均；这可以防止强化学习过程中响应长度增长过快。对于重要性采样，比率按 token 计算：\(r_{t,i}=\mathrm{sg}[\pi_{\theta}(o_{i,t})/\mu_{\theta_{\mathrm{old}}}(o_{i,t})]\)。每个 token 的训练概率由训练框架中的当前模型产生，推理概率则是该 token 生成时由 rollout 模型产生的概率；对于部分 rollout，我们不重新计算推理概率。我们使用四个解耦边界对重要性采样比率进行裁剪，正优势对应 \(\epsilon_{+}^{l},\epsilon_{+}^{h}\)，负优势对应 \(\epsilon_{-}^{l},\epsilon_{-}^{h}\)，得到裁剪掩码 \(M=\mathds{1}\big[(A\geq 0\wedge\epsilon_{+}^{l}\leq r\leq\epsilon_{+}^{h})\vee(A<0\wedge\epsilon_{-}^{l}\leq r\leq\epsilon_{-}^{h})\big]\)。正、负边界均初始化为 \([0.2,\,5.0]\)，并在运行时根据策略熵独立调整：当熵过低时，我们放宽正边界并收窄负边界，以将熵拉回正常范围；当熵过高时则相反。在整个训练过程中，我们还监控每个方向的 token 裁剪率。
We adopt Group Relative Policy Optimization (GRPO) in Eq. (1), with the advantage computation detailed in §4.3. For loss aggregation we use prompt-mean aggregation, which averages the surrogate at the prompt level rather than over all response tokens; this prevents response length from growing too quickly during RL. For importance sampling, the ratio is computed per token: \(r_{t,i}=\mathrm{sg}[\pi_{\theta}(o_{i,t})/\mu_{\theta_{\mathrm{old}}}(o_{i,t})]\). The training probability of each token is produced by the current model in the training framework, and the inference probability is the one produced by the rollout model when that token was generated; for partial rollouts we do not recompute inference probabilities. We clip the importance sampling ratio with four decoupled bounds, \(\epsilon_{+}^{l},\epsilon_{+}^{h}\) for positive advantages and \(\epsilon_{-}^{l},\epsilon_{-}^{h}\) for negative ones, giving the clip mask \(M=\mathds{1}\big[(A\geq 0\wedge\epsilon_{+}^{l}\leq r\leq\epsilon_{+}^{h})\vee(A<0\wedge\epsilon_{-}^{l}\leq r\leq\epsilon_{-}^{h})\big]\). Positive and negative bounds are both initialized to \([0.2,\,5.0]\) and tuned independently at runtime based on policy entropy: when entropy is too low, we widen the positive bounds and narrow the negative ones to pull entropy back to the normal range, and we do the opposite when entropy is too high. Throughout training we also monitor the token clipping rate of each direction.
我们从四个类别评估智能体式能力：代码智能体、网络安全、通用智能体和视觉智能体。我们的评估将公开基准与三个内部基准相结合：MiMo Code Bench、MiMo Cyber Bench 和 MiMo Visual Coding。
We evaluate agentic capabilities across four categories: code agent, cybersecurity, general agent, and visual agent. Our evaluation combines public benchmarks with three internal benchmarks: MiMo Code Bench, MiMo Cyber Bench, and MiMo Visual Coding.
我们使用 DeepSWE v1.1、ProgramBench 和 MiMo Code Bench 评估软件工程能力。DeepSWE 侧重于长时程开发任务。ProgramBench 评估端到端软件构建：智能体必须根据编译后的二进制文件和文档重建程序，产出能够复现参考程序行为的实现。MiMo Code Bench 提供了一项额外的内部评估，覆盖多样化的编码任务，用于评测编码智能体。
We evaluate software-engineering capabilities using DeepSWE v1.1, ProgramBench, and MiMo Code Bench. DeepSWE focuses on long-horizon development tasks. ProgramBench assesses end-to-end software construction: agents must reconstruct a program from its compiled binary and documentation, producing an implementation that reproduces the reference program's behavior. MiMo Code Bench provides an additional in-house evaluation of coding agents across a diverse range of coding tasks.
我们使用 CyberGym¹¹ ¹我们根据第 4.2.4 节所述方法修正了存在缺陷的评估环境。、ExploitGym、ExploitBench、SEC Bench Pro 和 MiMo Cyber Bench 评估漏洞复现与利用的互补方面。CyberGym 衡量智能体在真实软件中复现漏洞的能力，而 SEC Bench Pro 强调从缺陷报告中复现复杂漏洞。ExploitGym 评估超越单纯复现崩溃的漏洞利用开发，ExploitBench 则通过多个利用阶段及由此产生的安全影响来评估进展。MiMo Cyber Bench 以内部网络安全评估补充这些公开基准。
We evaluate complementary aspects of vulnerability reproduction and exploitation using CyberGym¹¹ ¹We corrected the flawed evaluation environments based on the method described in Section 4.2.4., ExploitGym, ExploitBench, SEC Bench Pro, and MiMo Cyber Bench. CyberGym measures agents' ability to reproduce vulnerabilities in real-world software, while SEC Bench Pro emphasizes reproducing complex vulnerabilities from bug reports. ExploitGym assesses exploit development beyond merely reproducing a crash, and ExploitBench evaluates progress through multiple exploitation stages and the resulting security impact. MiMo Cyber Bench supplements these public benchmarks with an in-house cybersecurity evaluation.
我们从工具使用、专业知识工作、基于终端的问题求解和计算机使用四个方面评估通用智能体。AutomationBench 在模拟 SaaS 环境中通过 REST API 评估跨应用工作流编排，包括 API 发现和对业务规则的遵循。Toolathlon-Verified 使用多样化工具评估长时程、多应用工作流，包括通过模型上下文协议（MCP）暴露的工具。GDPval-AA v2.1 是 Artificial Analysis 针对 GDPval 的评估框架，评估具有经济价值任务上专业交付成果的质量。JobBench 评估领域专家认定为高优先级、可委托给 AI 智能体的工作场所工作流。Agents' Last Exam 评估具有可验证结果的长时程、有经济价值的专业任务。Terminal-Bench 4.0 和 2.1 评估终端环境中复杂任务的完成情况。OSWorld-Verified 评估跨真实网页和桌面应用的交互式计算机使用。
We evaluate general-purpose agents across tool use, professional knowledge work, terminal-based problem solving, and computer use. AutomationBench evaluates cross-application workflow orchestration through REST APIs in simulated SaaS environments, including API discovery and adherence to business rules. Toolathlon-Verified evaluates long-horizon, multi-application workflows using diverse tools, including those exposed through the Model Context Protocol (MCP). GDPval-AA v2.1, Artificial Analysis's evaluation framework for GDPval, assesses the quality of professional deliverables on economically valuable tasks. JobBench evaluates workplace workflows that domain experts identify as high-priority for delegation to AI agents. Agents' Last Exam evaluates long-horizon, economically valuable professional tasks with verifiable outcomes. Terminal-Bench 4.0 and 2.1 evaluate the completion of complex tasks in terminal environments. OSWorld-Verified evaluates interactive computer use across real web and desktop applications.
我们使用 MiMo Visual Coding 评估视觉编码能力，这是一个内部开发的基准，涵盖开放式设计和高保真视觉复现。该基准包含 WebDev 和 Image2Code 等任务，分别涉及构建网站以满足用户请求，以及将参考图像转化为视觉上忠实的代码实现。
We evaluate visual coding capabilities using MiMo Visual Coding, an internally developed benchmark spanning open-ended design and high-fidelity visual replication. The benchmark includes tasks such as WebDev and Image2Code, which involve building websites to fulfill user requests and translating reference images into visually faithful code implementations, respectively.
所有可配置推理努力程度的基线模型均在支持的最高设置（max）下进行评估。
All baseline models with configurable reasoning effort are evaluated at the highest supported setting (max).
在训练过程中监控性能变化，以验证规模扩张后的强化学习算力。如图 9 所示，尽管不同检查点之间存在波动，Flash 与 Pro 在强化学习训练期间均在 DeepSWE v1.1、AutomationBench v1.0.6 和 MiMo Visual Coding 上取得了整体提升。这些增益通常伴随着总 token 数的增加，表明更强的任务性能与更多的 token 使用量同步发展。我们进一步研究了多 harness 训练的效果，结果如图 10 所示。专门的训练在四个训练用 mini-harness 以及三个留出 harness（codex、claude code 和 mini-swe-agent）上均提升了 DeepSWE v1.1 的性能。尽管检查点之间存在波动，三个留出 harness 在训练过程中均有提升，其平均 Pass@1 从约 50% 提高到 66%。训练 harness 与留出 harness 之间的平均性能差距也缩小了，这为所学编码能力能够跨 harness 实现迁移提供了证据。这些结果支持我们采用轻量、模块化的 mini-harness 在强化学习过程中引入受控多样性的设计。
The performance changes are monitored during the process to validate the scaled RL compute. As shown in Figure 9, both Flash and Pro achieve overall improvements on DeepSWE v1.1, AutomationBench v1.0.6, and MiMo Visual Coding during RL training, despite fluctuations between checkpoints. These gains generally accompany increasing total token counts, showing that stronger task performance develops alongside greater token usage. We further investigate the effect of multi-harness training, with results demonstrated in Figure 10. The dedicated training improves DeepSWE v1.1 performance across both the four training mini-harnesses and the three held-out harnesses: codex, claude code, and mini-swe-agent. Despite fluctuations between checkpoints, all three held-out harnesses improve over the course of training, with their mean Pass@1 increasing from approximately 50% to 66%. The gap between the mean performance on training and held-out harnesses also narrows, providing evidence that the learned coding capabilities transfer across harness implementations. These results support our design of using lightweight, modular mini-harnesses to introduce controlled diversity during RL.
在混合强化学习之后，我们使用多前缀多教师同策略蒸馏（Multi-Prefix Multi-Teacher On-Policy Distillation，MOPD2）来整合由不同任务（包括难以验证的任务）训练出的教师模型的能力。在 MiMo-V2-Flash 的 MOPD 基础上，我们在具备合适 mixRL 教师的领域保留学生自主 rollout（标准 MOPD），并新增以 prefix 为条件的单轮 rollout。图 13 展示了教师配置以及标准蒸馏与 prefix 条件蒸馏的工作流程。
After mixed RL, we use Multi-Prefix Multi-Teacher On-Policy Distillation (MOPD2) to combine capabilities from teachers trained for different tasks, including tasks that are hard to verify. Building on MOPD from MiMo-V2-Flash, we retain autonomous student rollouts in domains with suitable mixRL teachers (Standard MOPD) and add prefix-conditioned single-turn rollouts. Figure 13 illustrates the teacher configurations and the standard and prefix-conditioned distillation workflows.
前缀来自教师 rollout（Teacher-Prefix OPD）或 SFT 数据（SFT-Prefix OPD）。一条包含 \(k\) 个 assistant 轮次的轨迹可提供 \(k\) 个完整的历史前缀，每个前缀均截止于对应轮次之前。学生从每个前缀出发采样一个新轮次，而不重新生成此前的交互。预先指定的教师以相同历史及学生此前 token 为条件，提供 token 级监督。
Prefixes come from teacher rollouts (Teacher-Prefix OPD) or SFT data (SFT-Prefix OPD). A trajectory with \(k\) assistant turns provides \(k\) complete history prefixes, each ending before the corresponding turn. The student samples one new turn from each prefix without regenerating earlier interactions. A preassigned teacher provides token-level supervision conditioned on the same history and the student's preceding tokens.
对于难以设计可靠 RL 奖励的开放域任务，我们在高质量合成示范上训练 SFT 教师。其训练对长程任务中学生反复偏离后所到达的历史覆盖可能有限。因此，SFT-Prefix OPD 让每次 rollout 都从固定的示范前缀开始，从而限制采样轮次之前的偏离。示范提供上下文，而学生生成自己的续写，而非模仿固定回复。
For open-domain tasks where reliable RL rewards are difficult to design, we train SFT teachers on high-quality synthetic demonstrations. Their training may provide limited coverage of histories reached after repeated student deviations in long-horizon tasks. SFT-Prefix OPD therefore starts each rollout from a fixed demonstration prefix, limiting deviations before the sampled turn. Demonstrations supply the context, while the student generates its own continuation rather than imitating a fixed response.
MOPD2 进一步将经 RL 训练的模型的能力扩展到可靠训练时验证具有挑战性的领域，包括环境复杂或验证器难以设计的领域，例如长程游戏开发、科学研究和具身智能。MiMo-V2.6 的最终评测结果见表 5.6。MiMo-V2.6 系列相较 MiMo-V2.5 取得显著提升，在各领域达到与前沿模型相当的性能。
MOPD2 further extends the capabilities of the RL-trained model to domains where reliable training-time verification is challenging, including those with complex environments or verifiers that are difficult to design, such as long-horizon game development, scientific research, and embodied intelligence. The final evaluation results of MiMo-V2.6 are reported in Table 5.6. The MiMo-V2.6 series delivers substantial improvements over MiMo-V2.5, achieving performance comparable to that of frontier models across various domains.
MiMo-V2.6 系列将强化学习（RL）与 OPD 训练扩展到由智能体式 rollout 构成的大规模混合任务批次。为支持灵活的智能体式 rollout 场景，我们定义了执行模型与轨迹数据结构，并通过惩罚模块（Penalty Module，§6.1）精化学习信号。在大批量规模下，我们实现了 Harness Pool 以承载多个 harness 并维持高 rollout 并发度，同时实现 Payload Porter 以缓冲数万条带有大量路由与多模态数据的轨迹（§6.2）。我们实现了 Sample Mixer，它与动态采样器和部分 rollout 协同工作，稳定地交付符合指定训练分布的训练批次（§6.3）。为实现稳定高效的 RL 训练，我们在训练引擎与推理引擎之间对齐 MoE 路由与 top-p 采样候选集，同时针对 RL 工作负载对两个引擎进行优化（§6.4）。
The MiMo-V2.6 series scales RL and OPD training to large mixed-task batches of agentic rollouts. To support flexible agentic rollout scenarios, we define the execution model and trajectory data structure, and refine the learning signal with a Penalty Module (§6.1). At large batch sizes, we implement the Harness Pool to host multiple harnesses and sustain high rollout concurrency, and the Payload Porter to buffer tens of thousands of trajectories heavy with routing and multimodal data (§6.2). We implement a Sample Mixer that works with the dynamic sampler and partial rollout to stably deliver training batches matching a specified training distribution (§6.3). Towards stable and efficient RL training, we align MoE routing and top-p sampling candidate sets between the training and inference engines, while optimizing both engines for RL workloads (§6.4).
智能体式 rollout 跨越多个轮次与对话上下文，而结果奖励只能提供粗粒度监督。因此，我们采用以智能体为中心的执行模型，并将轨迹数据组织为层次结构。在该层次结构中，一个可配置的惩罚模块（Penalty Module）施加损失掩码与优势塑形。它针对局部模型错误，并将基础设施故障排除在训练信号之外。
Agentic rollouts span multiple turns and dialogue contexts, while outcome rewards provide only coarse supervision. We therefore adopt an agent-centric execution model and organize trajectory data as a hierarchy. Within this hierarchy, a configurable Penalty Module applies loss masking and advantage shaping. It targets local model errors and keeps infrastructure failures out of the training signal.
我们将 rollout 从以推理为中心的设计转向以智能体为中心的设计：每个序列作为一个 Agent Loop 运行，它拥有环境生命周期、管理其产生的对话，并按需调用推理引擎。该生命周期涵盖初始化、交互、奖励评估（例如运行测试用例）以及清理。在交互过程中，该循环暴露一个请求端点，外部智能体通过调用它来驱动 rollout。每个对话同时以字符串前缀和 token 序列两种形式保存。前缀匹配用于定位传入请求所扩展的对话，只有新的后缀会被 tokenize 并传给推理引擎，从而使其接口保持纯粹的 token-in、token-out。
We shift rollout from an inference-centric design to an agent-centric one: each sequence runs as an Agent Loop that owns the environment lifecycle, manages the dialogues it produces, and calls the inference engine on demand. The lifecycle covers setup, interaction, reward evaluation (e.g., running test cases), and cleanup. During interaction the loop exposes a request endpoint, and the external agent drives the rollout by calling it. Each dialogue is kept both as a string prefix and as a token sequence. Prefix matching locates the dialogue that an incoming request extends, and only the new suffix is tokenized and passed to the inference engine, keeping its interface purely token-in, token-out.
子智能体、上下文压缩以及多种智能体角色会在一次 rollout 内产生并发的对话分支。我们将轨迹数据组织为四级层次结构：Sample \(\rightarrow\) Sequence \(\rightarrow\) Context \(\rightarrow\) Segment。Sample 是由 Sample Mixer（§6.3）分发的提示。在 GRPO 等组式算法中，一个 Sample 会生成一组 Sequence，该组作为一个整体被接受或拒绝。Sequence 是一次 Agent Loop 执行，可能包含多个并发的 Context。Context 是一个对话分支，持有一个 Segment 列表；它是前缀匹配、KV-cache 复用和训练数据导出的单位。Segment 是单个轮次——一条系统或用户消息、一次模型生成，或一个工具结果——只有模型生成的轮次才贡献损失。
Subagents, context compaction, and multiple agent roles produce concurrent dialogue branches within one rollout. We organize the trajectory data into a four-level hierarchy: Sample \(\rightarrow\) Sequence \(\rightarrow\) Context \(\rightarrow\) Segment. A Sample is a prompt dispatched by the Sample Mixer (§6.3). In group-wise algorithms such as GRPO, the Sample spawns a group of Sequences, and the group is accepted or rejected as a whole. A Sequence is one Agent Loop execution and may contain several concurrent Contexts. A Context is one dialogue branch holding a list of Segments; it is the unit of prefix matching, KV-cache reuse, and training-data export. A Segment is a single turn—a system or user message, a model generation, or a tool result—and only model-generated turns contribute to the loss.
组式算法将结果奖励均匀分摊到一个序列的所有模型生成 token 上。在上述层次结构中，信用分配很少是均匀的：某些轮次偏离路径或退化，某些失败与模型无关。为了将信用分配到应得之处，我们将检测与其对训练的影响分离。Rule 通过手工逻辑或基于模型的评判器来判定一个 segment、context 或 sequence。例如，规则可以捕获不可归因于模型的基础设施故障、乱码 token 模式、对不可用工具的调用以及重复。Strategy 将动作绑定到层次结构的某一层级：mask 将命中的内容排除出损失，advantage shaping 对命中 token 上的优势进行设置、缩放或减去，monitor 仅记录指标。复合的 early stop 策略在其规则触发时立即停止 rollout，将结果奖励置零，对触发轮次和更早轮次分别施加不同动作，并掩码同级上下文。惩罚沿层次结构逐级升级：没有存活模型轮次的 context 被丢弃，没有存活 context 的 sequence 获得零优势，没有存活 sequence 的 sample 被拒绝。训练中应用的具体惩罚见 §4.3.3。
Group-wise algorithms spread the outcome reward evenly across all model-generated tokens of a sequence. Within the hierarchy above, credit is rarely uniform: some turns are off-path or degenerate, and some failures are unrelated to the model. To assign credit where it is due, we separate detection from its effect on training. A Rule judges a segment, context, or sequence by handcrafted logic or a model-based judge. For example, rules can catch infrastructure failures not attributable to the model, garbled token patterns, calls to unavailable tools, and repetition. A Strategy binds an action to a level of the hierarchy: mask excludes the hit content from the loss, advantage shaping sets, scales, or subtracts the advantages on hit tokens, and monitor records metrics alone. The composite early stop strategy halts the rollout as soon as its rule fires, zeroes the outcome reward, applies separate actions to the triggering turn and earlier turns, and masks sibling contexts. Penalties escalate along the hierarchy: a context with no surviving model turns is dropped, a sequence with no surviving context receives zero advantage, and a sample with no surviving sequence is rejected. The specific penalties applied in training are described in §4.3.3.
将强化学习训练扩展到大批量会提高 rollout 并发度，也会增加轨迹数据的内存与通信开销。混合任务批次又引入了另一个异构维度：单个批次混合了多个 harness，而每个训练样本组绑定到其中一个 harness。在执行侧，Harness 池将并发的 harness 实例与 Agent Loop 托管在持久化的多租户 actor 池中，harness 代码库、智能体行为与环境设置均可独立配置。在数据侧，Payload Porter 使 driver 仅基于轻量级元数据进行调度。重型 payload 只写入分布式存储一次，并在消费处就地打包。多模态 payload 同样按元数据与像素拆分，并采用增量传输与负载均衡编码。
Scaling RL training to large batches increases rollout concurrency and the memory and communication costs of trajectory data. Mixed-task batches add another axis of heterogeneity: a single batch mixes several harnesses, with each training sample group bound to one harness. On the execution side, the Harness Pool hosts concurrent harness instances and Agent Loops in persistent multi-tenant actor pools, and harness codebases, agent behavior, and environment settings are configured independently. On the data side, the Payload Porter keeps the driver scheduling on lightweight metadata alone. Heavy payloads are written once to a distributed store and packed where they are consumed. Multi-modal payloads split the same way into metadata and pixels, with incremental transport and load-balanced encoding.
我们使用 Ray actor 在集群上执行智能体 harness 与 Agent Loop。若为每个 harness 实例和 Agent Loop 都分配专用 Ray actor，则每个 actor 会在 Ray 全局控制存储（GCS）节点上占用一个文件描述符，大批量会耗尽 GCS 节点的文件描述符。因此我们改为运行固定大小的持久化 host actor 池，每个 host 承载多个并发租户——模型侧的 Agent Loop 与环境侧的 harness 实例。每个租户维护自己的轨迹状态，并按在途实例数进行均衡分配。一个 host 是单个进程，其租户共享一个事件循环。在模型侧，它们还共享一个请求端点、一个推理代理和一个分词器；在环境侧，共享一个已导入的 harness 代码库。共享事件循环会让一个阻塞调用拖住所有租户，因此阻塞性工作——环境操作与分词——放到后台线程执行。这样可将进程与服务开销分摊到并发 rollout 上，从而在不按比例增加 actor 数量的情况下支持更大的并发批次。
We use Ray actors to execute agent harnesses and Agent Loops across the cluster. A dedicated Ray actor for every harness instance and Agent Loop would cost one file descriptor per actor on Ray's global control store (GCS) node, and a large batch would exhaust the GCS node's file descriptors. We instead run fixed-size pools of persistent host actors, each carrying many concurrent tenants—Agent Loops on the model side and harness instances on the environment side. Each tenant keeps its own trajectory state, and assignments are balanced by in-flight instance count. A host is a single process whose tenants share one event loop. On the model side they also share one request endpoint, one inference proxy, and one tokenizer; on the environment side, one imported harness codebase. A shared event loop would let one blocking call stall every tenant, so blocking work—environment operations and tokenization—runs on background threads. This amortizes process and service overhead across concurrent rollouts, supporting larger concurrent batches without a proportional increase in actor count.
我们分别配置 harness 代码库、智能体行为与环境设置，以在单次训练运行中适配多样化的任务。不同数据源可以使用不同代码库，而智能体配置在同一数据源内可随 prompt 变化。每个训练样本组使用统一配置以进行组相对优势估计。一个进程只能导入一个 harness 代码库，因此不同代码库运行在独立的池中。这些池从固定的 host actor 预算中获得可配置的份额——该预算在启动时设定一次，与每步调度的训练数据混合无关。由此，harness 多样性与 rollout 并发度可在同一执行框架内扩展。
We configure harness codebases, agent behavior, and environment settings separately to accommodate diverse tasks within a single training run. Different data sources can use different codebases, while agent configurations vary across prompts within a source. Each training sample group uses a common configuration for group-relative advantage estimation. One process can import only one harness codebase, so different codebases run in separate pools. These pools receive configurable shares of a fixed budget of host actors—set once at startup, independent of the training-data mixture that is scheduled every step. Harness diversity and rollout concurrency thus scale within one execution framework.
扩展后的批次必须一次缓冲数万条序列，且每条序列都很重：除 token id 与对数概率外，还携带 MoE 路由数据、top-p 采样索引与多模态数据。Payload 体量随序列数量与长度增长，若将所有 payload 汇聚到单个 driver 节点，批次大小将受限于该节点内存。因此我们将数据面与控制面解耦，在 rollout 结束时拆分每条序列：其 payload 只写入一次分布式键值存储（例如 Ray object store 或 TransferQueue），而 driver 仅基于轻量级元数据执行全部调度——标量奖励、各上下文长度以及寻址各 payload 的键。组完成时，仅从存储读取所需字段：accept 时 hook 所需的少数几列。若配置了组级评分器，则它与 Agent Loop 完全异步运行——其延迟被隐藏，结果可自由滞后——并在返回时重写组奖励。随后采样器仅依据元数据按 passrate 接受或拒绝该组，hook 施加长度惩罚、计算组相对优势并应用优势塑形。逐 token 优势被写回存储；优势全为零的组默认被丢弃。在 OPD 下，奖励评估由教师评分替代：每条轨迹发送至教师服务器，其分数被异步收集到同一分布式存储中。在批次产出时——即每个数据源已贡献其批次份额后——yield hook 将接受的序列打包为 micro-batch 并分配到各 rank，全程不触碰张量。打包时，每个训练张量并行（TP）组由一个 packer 服务该组内所有 rank。它从存储中未填充的行里仅取上下文并行（CP）窗口所触及的行，并只切出该窗口。结果以单个只读内存副本在 TP 组内共享。这避免了 driver 上的全批次聚合以及打包期间的稠密填充中间量。
Scaled batches must buffer tens of thousands of sequences at a time, and each is heavy: besides token ids and log-probabilities, it carries MoE routing data, top-p sampling indices, and multimodal data. Payload volume grows with both sequence count and length, and gathering every payload on one driver node ties batch size to that node's memory. We therefore disaggregate the data plane from the control plane, splitting each sequence at rollout finish: its payload is written once into a distributed key-value store (e.g., the Ray object store or TransferQueue), while the driver runs all scheduling on lightweight metadata—scalar rewards, per-context lengths, and the keys addressing each payload. At group finish, only the fields needed are read from the store: a few columns for the accept-time hook. A group-wise grader, when configured, runs fully asynchronously alongside the Agent Loops—its latency hidden and its results free to lag—and rewrites the group rewards on return. The sampler then accepts or rejects the group by passrate, on metadata alone, and the hook imposes length penalties, computes group-relative advantages, and applies advantage shaping. The per-token advantages are written back into the store; groups whose advantages are all zero are dropped by default. Under OPD, reward evaluation is replaced by teacher scoring: each trajectory is sent to a teacher server, and its scores are collected asynchronously into the same distributed store. At batch yield—once each data source has contributed its share of the batch—a yield hook packs the accepted sequences into micro-batches and assigns them to ranks, touching no tensor. At pack time, one packer per training tensor-parallel (TP) group serves every rank in the group. From the unpadded rows in the store, it fetches only those its context-parallel (CP) window touches and cuts out that window alone. The result is shared across the TP group as a single read-only in-memory copy. This avoids full-batch aggregation on the driver and dense padded intermediates during packing.
多模态 payload 遵循相同的元数据/payload 拆分，但需要额外注意：随着智能体在多轮中反复截图并读取图像，单条轨迹可累积数 GB 的此类数据——存储代价高，且随历史增长重传代价也高。因此在 rollout 期间，Agent Loop 仅传输请求之间的多模态增量（§6.4）。训练时，每个图像项都必须经过视觉编码器。由于编码器在张量并行组内复制，而 LLM 主干是分片的，编码首先以数据并行方式运行——图像项在各 rank 间均衡分配，与各序列 token 落在何处无关。编码后，嵌入被重分发到持有对应 token 的 rank。payload 本身留在分布式键值存储中：负载均衡规划只读取项元数据，像素仅在编码器计算时获取。这在适配不均衡多模态工作负载的同时，限制了冗余的 payload 移动。
Multi-modal payloads follow the same meta/payload split, but demand extra care: as the agent repeatedly takes screenshots and reads images across turns, a single trajectory can accumulate gigabytes of such data—costly to store, and costly to retransmit as the history grows. During rollout, the Agent Loop therefore ships only the multi-modal delta between requests (§6.4). For training, every image item must pass through the vision encoder. Because the encoder is replicated across the tensor-parallel group while the LLM backbone is sharded, encoding runs data-parallel first—image items are balanced across ranks independently of where each sequence's tokens land. After encoding, embeddings are redistributed to the ranks holding the corresponding tokens. The payload itself stays in the distributed key-value store: load-balance planning reads only item metadata, and pixels are fetched only for the encoder computation. This limits redundant payload movement while accommodating uneven multi-modal workloads.
自 MiMo-V2-Flash 以来，我们一直维护一个 Data Scheduler，它以跨数据源的指定训练分布为目标，并配合动态采样器和部分 rollout。混合任务强化学习必须在 rollout 时长和过滤率存在巨大差异的情况下仍保持这一分布，从而使每个数据源都得到有效训练。在 25 个已分析的数据源中，平均生成 token 数和活跃 rollout 时长分别相差 \(90\times\) 和 \(66\times\)（图 15），这促使我们采用能适应各数据源工作负载的调度方式。为应对这些挑战，我们实现了 Sample Mixer，通过四种机制来填充指定的训练分布：自适应 Rollout 并发度设置各数据源的预算，自适应 Rollout 调度在预算内选择数据源，预测式 Rollout 分发将新的 rollout 分配到各 rank，样本回放则覆盖启动与恢复阶段。
Since MiMo-V2-Flash, we have maintained a Data Scheduler that targets a specified training distribution across data sources, together with a dynamic sampler and partial rollout. Mixed-task RL must preserve this distribution despite large variations in rollout duration and filtering rates, so that every data source is trained effectively. Across 25 profiled data sources, the mean generated tokens and the active rollout duration vary by \(90\times\) and \(66\times\), respectively (Figure 15), motivating scheduling that adapts to each source's workload. To meet these challenges, we implement the Sample Mixer, with four mechanisms filling the specified training distribution: Adaptive Rollout Concurrency sets per-source budgets, Adaptive Rollout Scheduling selects data sources within the budgets, Predictive Rollout Dispatch places new rollouts across ranks, and Sample Replay covers startup and recovery.
较慢的数据源需要更多并发 rollout，才能维持相同的训练贡献。对于数据源 \(i\)，令 \(B_{i}\) 表示每个训练步保留的样本组目标数量，\(r_{i}\) 表示估计的组接受率，\(t_{i}\) 表示估计的活跃 rollout 时长，其中包括模型生成和环境交互，但不包括训练步之间的暂停。期望生成需求为 \(m_{i}=B_{i}/r_{i}\)；在固定训练吞吐下，所需并发度随 \(t_{i}m_{i}\) 缩放。我们为每个数据源分配 \((1+p_{i})m_{i}\) 组的调度预算，其中过采样比 \(p_{i}\) 满足
Slower sources need more concurrent rollouts to sustain the same training contribution. For source \(i\), let \(B_{i}\) denote the target number of retained sample groups per training step, \(r_{i}\) the estimated group acceptance rate, and \(t_{i}\) the estimated active rollout duration, including model generation and environment interaction but excluding pauses between training steps. The expected generation demand is \(m_{i}=B_{i}/r_{i}\); at a fixed training throughput, the required concurrency scales with \(t_{i}m_{i}\). We assign each source a scheduling budget of \((1+p_{i})m_{i}\) groups, where the oversampling ratio \(p_{i}\) satisfies
共享因子 \(c\) 使 \(p_{i}\) 的需求加权均值保持在全局过采样比 \(\bar{p}\)，并受各数据源上下界约束。随着工作负载变化，我们根据近期的时序和接受率统计重新计算该分配。在稳态下，并发需求随数据源的 rollout 时长和目标增大而增大，随其接受率升高而减小。
The shared factor \(c\) keeps the demand-weighted mean of \(p_{i}\) at the global oversampling ratio \(\bar{p}\), subject to per-source bounds. We recompute this allocation from recent timing and acceptance statistics as workloads change. In steady state, the concurrency requirement grows with a source's rollout duration and target, and falls with its acceptance rate.
在这些预算内，调度在长期生成需求与当前训练批次的推进之间取得平衡。若当前批次已从数据源 \(i\) 接受 \(A_{i}\) 组，则其调度权重为
Within these budgets, scheduling balances long-run generation demand with progress toward the current training batch. If \(A_{i}\) groups have already been accepted from source \(i\) for the current batch, its scheduling weight is
其中 \((x)^{+}=\max(x,0)\)，\(\alpha\in[0,1]\)。目标项维持生成需求；亏空项优先考虑仍有剩余亏空的数据源。这些权重驱动平滑加权轮询；稳态启动将 \(\alpha=0.5\) 与按 \(t_{i}m_{i}\) 比例分配的初始并发度相结合。图 16 中的轨迹驱动仿真在固定并发上限且数据源预算不构成约束的条件下比较了各策略：在该工作负载下，亏空校正调度（\(\alpha=0.5\)）相比基于亏空的调度（\(\alpha=0\)）提升了占用稳定性，相比基于目标的调度（\(\alpha=1\)）提升了采集均衡性。仿真时长按图 15 中 End 处各数据源的平均 rollout 时间进行校准。
where \((x)^{+}=\max(x,0)\) and \(\alpha\in[0,1]\). The target term maintains generation demand; the deficit term prioritizes sources with a remaining deficit. These weights drive smooth weighted round-robin; steady-state startup combines \(\alpha=0.5\) with initial concurrency allocated in proportion to \(t_{i}m_{i}\). The trace-driven simulation in Figure 16 compares the policies at a fixed concurrency limit with nonbinding source budgets: Deficit-corrected scheduling (\(\alpha=0.5\)) improves occupancy stability over Deficit-based scheduling (\(\alpha=0\)) and collection balance over Target-based scheduling (\(\alpha=1\)) in this workload. Simulation durations are calibrated to each source's mean rollout time at End in Figure 15.
我们联合估计 KV 需求和预期推理并发度，以指导新 rollout 在各 rank 上的准入与放置。按来源的先验估计一个 rollout 的总序列长度（输入 token 与生成 token），并在各上下文上求和；这些先验由已完成的 rollout 更新，并用于估计 KV 需求。仅当估计 KV 需求的安全裕度倍数能够容纳于目标 rank 的剩余 GPU KV 容量时，我们才准入该 rollout。分层缓存（§6.4）横跨 HBM 与固定主机内存池：这两层共同保留已准入 rollout 的状态。rollout 时间中用于环境执行的估计比例，将 rollout 并发度转换为预期推理并发度。我们用 CUDA 图捕获所用的最大运行请求数来约束该期望值，从而限制那些会延长 rollout 生命周期并加剧陈旧性的排队延迟。为在两种约束下最大化吞吐量，一种贪心启发式选择剩余容量最大的可行 rank——剩余容量取可用并发槽位与剩余 KV 容量中的较小者，二者均以序列为单位表示——且这两个量在每次放置后都会更新。
We jointly estimate KV demand and expected inference concurrency to guide the admission and placement of new rollouts across ranks. Per-source priors estimate a rollout's total sequence length (input and generated tokens), summed across contexts; these priors are updated from completed rollouts and used to estimate KV demand. We admit a rollout only when a safety-margin multiple of its estimated KV demand fits within the target rank's remaining GPU KV capacity. Hierarchical caching (§6.4) spans HBM and a pinned host pool: the two tiers jointly retain the state of admitted rollouts. The estimated fraction of rollout time spent in environment execution converts rollout concurrency into expected inference concurrency. We bound this expectation by the max running requests used for CUDA graph capture, limiting queueing delays that prolong rollout lifetimes and increase staleness. To maximize throughput under both constraints, a greedy heuristic selects the feasible rank with the greatest remaining capacity—the minimum of available concurrency slots and remaining KV capacity, both expressed in sequence units—with both quantities updated after each placement.
我们使用样本回放来加速从慢速来源进行的初始收集。在性能剖析中，启动阶段的样本收集耗时约为持续运行的 \（1.8\times）。在慢速来源内部，较短的 rollout 往往先完成，即使满足了各来源的目标，也会使初始批次产生偏差。因此，在启动或检查点恢复后的第一个收集步骤中，我们复用来自选定慢速来源的已完成 rollout 组，在当前过滤规则下填补各来源剩余的缺口。全新启动回放假设存储的 rollout 是在本次运行的起始策略下生成的；恢复回放则复用恢复前生成、且仍处于各来源陈旧性限制内的已完成 rollout。将回放限制在第一个收集步骤，可减少对慢速来源的等待，同时保持指定的训练分布。
We use sample replay to accelerate initial collection from slow sources. In profiling, startup sample collection took approximately \(1.8\times) as long as that of continuous operation. Within slow sources, shorter rollouts tend to finish first, biasing the initial batch even when per-source targets are met. We therefore reuse completed rollout groups from selected slow sources during the first collection step after startup or checkpoint recovery, filling remaining per-source deficits under current filtering rules. Fresh-start replay assumes that stored rollouts were generated under the run's starting policy; recovery replay reuses completed rollouts generated before recovery and still within each source's staleness limit. Restricting replay to the first collection step reduces the wait for slow sources while preserving the specified training distribution.
我们扩展了 MiMo-V2-Flash 的强化学习与 OPD 基础设施，分别保留 SGLang 与 Megatron-LM 作为推理引擎与训练引擎。在 MiMo-V2.6 中，我们在 rollout 期间使用 MXFP4 作为专家（experts）的数据类型。我们在混合专家（MoE）路由与概率归一化上保持两个引擎之间的一致性。在推理侧，Context Cache 跨轮次携带路由记录、采样候选集与视觉输入以及 KV 状态，并将空闲状态卸载到主机内存。一个在 RL rollout 日志上训练的草稿模型通过投机解码进一步加速生成。在训练侧，我们降低了长序列的内存与通信开销。
We extend the RL and OPD infrastructure of MiMo-V2-Flash, keeping SGLang and Megatron-LM as the inference and the training engine, respectively. In MiMo-V2.6, we use MXFP4 as the experts' data type during rollout. We maintain alignment between the two engines in mixture-of-experts (MoE) routing and probability normalization. On the inference side, the Context Cache carries routing records, sampling candidate sets, and visual inputs alongside the KV state across turns, and offloads idle state to host memory. A draft model trained on RL rollout logs further accelerates generation through speculative decoding. On the training side, we reduce the memory and communication costs of long sequences.
为支持 MiMo-V2.6 系列，我们在每次参数更新后对专家应用量化–反量化（QDQ）。量化遵循 rollout 期间所用 MXFP4 Humming GEMM 内核的数值约束，因此两个引擎看到的专家权重完全一致。即使在参数完全相同的情况下，引擎之间的数值差异也可能翻转离散的专家选择；Rollout Routing Replay（R3）记录 rollout 期间使用的专家索引，并在训练期间重放它们，从而复现所捕获的执行路径。Top-k 与 top-p 采样在受限的候选集而非完整词表上重新归一化；我们在 rollout 期间记录每个 token 的候选集，并在其中对训练对数概率重新归一化，以匹配采样器的归一化方式。对于 top-p 采样，只有 GPU–CPU 传输是稠密的：我们传输一个固定形状、覆盖完整词表宽度的位图。固定形状避免了 GPU–CPU 同步；完整宽度即使面对跨越整个词表的候选集也绝不截断。后续所有阶段都是稀疏的：在典型的 top-p 为 0.97 时，候选集平均少于五个 token。R3 与 top-p 候选集重放的开销均可忽略不计，因为负载保持紧凑且位于关键路径之外。
To support the MiMo-V2.6 series, we apply quantize–dequantize (QDQ) to the experts after each parameter update. The quantization follows the numeric constraints of the MXFP4 Humming GEMM kernels used during rollout, so the two engines see identical expert weights. Even under identical parameters, numerical differences between the engines can flip discrete expert selections; Rollout Routing Replay (R3) records the expert indices used during rollout and replays them during training, reproducing the captured execution path. Top-k and top-p sampling renormalize over a restricted candidate set rather than the full vocabulary; we record each token's candidate set during rollout and renormalize the training log-probabilities within it, matching the sampler's normalization. For top-p sampling, only the GPU–CPU transfer is dense: we ship a bitmap of fixed shape and full-vocabulary width. The fixed shape avoids a GPU–CPU synchronization; the full width never truncates even a set that spans the whole vocabulary. All later stages are sparse: at a typical top-p of 0.97, a candidate set averages fewer than five tokens. Both R3 and top-p candidate-set replay incur negligible overhead, as the payloads stay compact and move off the critical path.
沿用 MiMo-V2-Flash 的请求级 KVCache，每个对话上下文都有一个持久键：在同一个策略版本内，后续轮次命中缓存的 KV——包括已生成的 token——并且只对新后缀进行 prefill。上下文缓存是一个有意为之的有状态选择：缓存状态的生命周期跨越每一轮次。因此，专家索引与候选集不在中间轮次返回，而只在 rollout 最终收集时返回，从而避免碎片化的逐轮通信与处理。历史多模态输入既不需要重新哈希也不需要重新发送：它们的视觉 token 已经位于缓存的 KV 之中。只有新引入的图像才跨越进程边界；完整视觉输入仅在策略更新或缓存未命中之后才发送。多轮环境交互将一条轨迹的实际耗时划分为 GPU 时间（生成）与工具时间（等待环境）；缓存的分层扩展在空间上反映了这一时间划分——状态在 GPU 时间驻留于 HBM，在工具时间驻留于固定（pinned）主机内存池。卸载与恢复在旁路 CUDA 流而非计算流上运行，绝不阻塞生成。由此 HBM 尽可能用于运行中的请求：解码批次越大，算术强度与算力利用率越高。
Following the request-level KVCache of MiMo-V2-Flash, each dialogue context has a persistent key: within one policy version, later turns hit the cached KV—including generated tokens—and prefill only the new suffix. Context caching is a deliberately stateful choice: the cached state outlives each turn. Expert indices and candidate sets are therefore not returned on intermediate turns but only when the rollout is finally collected, avoiding fragmented per-turn communication and processing. Historical multimodal inputs need neither re-hashing nor re-sending: their visual tokens already sit inside the cached KV. Only newly introduced images cross the process boundary; full visual inputs are sent only after a policy update or cache miss. Multi-turn environment interaction divides a trajectory's wall clock into GPU time (generation) and tool time (waiting on the environment); a hierarchical extension of the cache reflects this temporal split spatially—a state resides in HBM during GPU time and in a pinned host pool during tool time. Offloading and restoration run on side CUDA streams rather than the compute stream, never stalling generation. HBM is thereby devoted to running requests as far as possible: the larger the decode batch, the higher the arithmetic intensity and compute utilization.
RL rollout 默认使用 block-\(6\) DFlash 进行投机解码，取代从 SFT 继承的多 token 预测（MTP-\(3\)）配置。DFlash 最初在 SFT 策略上训练，随后在重采样以匹配 RL 训练分布的早期 RL rollout 日志上进行微调。在该默认配置下，平均接受长度比 MTP 配置高 31.3%。在较大的 RL 批次规模下，我们基于端到端吞吐而非仅接受率来选择草稿块大小。更小的草稿块减少验证工作：在混合任务 RL 评估中，block-\(6\) 相比 block-\(8\) 将全局平均吞吐提升约 6%，而平均接受长度变化很小。我们还应用低精度草稿计算（FP8）以降低草稿开销。在我们的长上下文工作负载上，经 RL 适配的 FP8 DFlash 相比基线实现约 10.3% 更高的单节点吞吐。
RL rollouts use block-\(6\) DFlash for speculative decoding by default, replacing the multi-token prediction (MTP-\(3\)) configuration inherited from SFT. Initially trained on the SFT policy, DFlash is then finetuned on early RL rollout logs resampled to match the RL training distribution. Under this default configuration, average accepted length is 31.3% higher than with the MTP configuration. At large RL batch sizes, we select the draft block size based on end-to-end throughput rather than acceptance rate alone. Smaller draft blocks reduce verification work: in mixed-task RL evaluation, block-\(6\) improves global average throughput by approximately 6% over block-\(8\), with little change in average accepted length. Low-precision draft computation (FP8) is also applied to reduce drafting overhead. On our long-context workload, RL-adapted FP8 DFlash achieves approximately 10.3% higher per-node throughput than the baseline.
我们以 1M token 的上下文长度训练 RL。MiMo-V2.6 将 128-token 滑动窗口层与全注意力机制交替排列。在上下文并行下，滑动窗口层只交换其查询能够触及的 KV——至多一个窗口大小的片段——因此每层流量受窗口而非序列长度约束。长序列还会膨胀内存占用，而 MoE 专家不均衡进一步推高峰值内存。优化器状态保留在 CPU 内存中，仅在参数更新时复制回来。所有损失计算融合进一个内核：策略梯度损失（带或不带 top-p 重新归一化）与 OPD 损失，可选地附带熵、标签 logit 与 top-p 质量等指标。该融合节省内存并缩短步进时间。
We train RL at a 1M-token context length. MiMo-V2.6 interleaves 128-token sliding-window layers with full attention. Under context parallelism, the sliding-window layers exchange only the KV their queries can reach—at most a window-sized segment—so per-layer traffic is bounded by the window rather than the sequence length. Long sequences also inflate the memory footprint, and MoE expert imbalance pushes peak memory higher. Optimizer states stay in CPU memory and are copied back only for parameter updates. All loss computation is fused into one kernel: the policy-gradient loss (with or without top-p renormalization) and the OPD loss, optionally with metrics such as entropy, label logit, and top-p mass. The fusion saves memory and reduces step time.
MiMo-V2.6 系列展示了强化学习在显著提升智能体式能力方面的潜力。要在这一进展的基础上继续前进，需要能力强的模型以及高质量的强化学习环境，这些环境应将有意义的任务与可靠的验证器配对。将这些资源与可复现的基线一同共享，有助于开源社区持续开展研究与创新。因此，我们开源了 MiMo-V2.6-Distill-Qwen-9B，这是一个从 MiMo 蒸馏得到的小模型，作为进一步强化学习训练的共享起点。除模型之外，我们还发布了高质量的强化学习环境和端到端的开源强化学习框架。在本节中，我们使用这些已发布的资源建立领域特定的 GRPO 基线。我们还以编码为案例，按照 MiMo-V2.6 所述的方法，开展了一项单独的多 harness 训练实验。我们的实验表明，在多个领域都取得了显著收益，证明了这些资源对强化学习训练的价值。
The MiMo-V2.6 series demonstrates the potential of reinforcement learning to substantially advance agentic capabilities. Building on this progress requires capable models and high-quality RL environments that pair meaningful tasks with reliable verifiers. Sharing these resources alongside reproducible baselines supports continued research and innovation in the open-source community. We therefore open-source MiMo-V2.6-Distill-Qwen-9B, a small model distilled from MiMo as a shared starting point for further RL training. Alongside the model, we release high-quality RL environments and an end-to-end open-source RL framework. In this section, we establish domain-specific GRPO baselines using these released resources. We also conduct a separate multi-harness training experiment using coding as a case study, following the approach described for MiMo-V2.6. Our experiments show substantial gains across multiple domains, demonstrating the value of these resources for RL training.
扩展强化学习的算力，并拓宽环境与测试框架（harness）的覆盖范围，能够为探索与学习创造更多机会。在这些设置中，有效的探索取决于协调动作并对环境反馈作出响应。因此，要释放智能体式强化学习（agentic RL）的潜力，首先需要具备扎实的任务求解能力基础。
Scaling RL computation and broadening the coverage of environments and harnesses create more opportunities for exploration and learning. Across these settings, effective exploration depends on coordinating actions and responding to environmental feedback. Realizing the potential of agentic RL therefore begins with a strong foundation of task-solving capabilities.
为向开源社区提供这样的基础，我们通过将 MiMo 的智能体式经验迁移到更小的模型，开发了 MiMo-V2.6-Distill-Qwen-9B。我们通过对 Qwen3.5-9B 在 MiMo 生成的数据上进行监督微调得到该模型，这些数据涵盖编程、通用领域、视觉与网络安全任务。该数据混合共包含 774 亿个 token，其中 272 亿个 loss token 参与 SFT 目标。表 4 总结了数据构成。
To provide the open-source community with such a foundation, we develop MiMo-V2.6-Distill-Qwen-9B by transferring MiMo's agentic experience into a smaller model. We obtain the model by supervised fine-tuning Qwen3.5-9B on MiMo-generated data spanning coding, general-domain, visual, and cybersecurity tasks. The mixture contains 77.4 billion total tokens, including 27.2 billion loss tokens that contribute to the SFT objective. Table 4 summarizes the data composition.
所得的 MiMo-V2.6-Distill-Qwen-9B checkpoint 在所报告的评测上均优于 Qwen3.5-9B（表 7.2）。例如，SWE-bench Pro 从 32.0 提升至 44.6，AutomationBench 从 5.0% 提升至 30.3%。我们将这些增强的任务求解能力视为通过强化学习进一步改进的可靠基础。因此，我们将该 checkpoint 用作下文所述领域特定 GRPO 实验与多测试框架编程实验的共同初始化。
The resulting MiMo-V2.6-Distill-Qwen-9B checkpoint improves over Qwen3.5-9B across the reported evaluations (Table 7.2). For example, SWE-bench Pro increases from 32.0 to 44.6, and AutomationBench from 5.0% to 30.3%. We view these strengthened task-solving capabilities as a promising foundation for further improvement through RL. We therefore use this checkpoint as the common initialization for the domain-specific GRPO experiments and the multi-harness coding experiment described below.
我们从同一个 MiMo-V2.6-Distill-Qwen-9B SFT 检查点出发，使用相应发布的开放环境，分别针对编程、网络安全、通用领域和视觉任务进行 GRPO 训练。除公开基准外，我们还在四个内部评测集上进行评估：用于软件工程的 MiMo Code Bench（mini）、用于漏洞复现的 MiMo Cyber Bench（mini）、用于知识工作的 MiMo General Bench（mini），以及用于网站开发的 MiMo Visual Coding（mini）。这些评测集遵循与其对应训练集相同的任务分布。
Starting from the same MiMo-V2.6-Distill-Qwen-9B SFT checkpoint, we conduct GRPO training separately for coding, cybersecurity, general-domain, and visual tasks using the corresponding released environments. Alongside public benchmarks, we evaluate on four internal sets: MiMo Code Bench (mini) for software engineering, MiMo Cyber Bench (mini) for vulnerability reproduction, MiMo General Bench (mini) for knowledge work, and MiMo Visual Coding (mini) for website development. These evaluation sets follow the same task distributions as their corresponding training sets.
表 7.2 中的编程结果通过单 harness 强化学习获得。我们在一个独立的编程实验中进一步研究多 harness 训练，同样从同一个 MiMo-V2.6-Distill-Qwen-9B SFT 检查点出发。遵循 MiMo-V2.6 中描述的多 harness 训练方法，我们在四个 mini-harness 上联合优化模型，并在这四个 harness 以及另外三个留出 harness 上进行评估。表 7.2 在三个编程评测和七个智能体 harness 上，比较了 Qwen3.5-9B 与多 harness 强化学习前后的 MiMo-V2.6-Distill-Qwen-9B。MiMo-V2.6-Distill-Qwen-9B 在全部 21 个数据集–harness 组合上均优于 Qwen3.5-9B，而多 harness 强化学习进一步提升了每一个组合。在 MiMo Code Bench（mini）上，相对于 SFT 的额外增益在七个 harness 上介于 1.8 到 9.3 个百分点之间。
The coding results in Table 7.2 are obtained through single-harness RL. We further investigate multi-harness training in a separate coding experiment, starting from the same MiMo-V2.6-Distill-Qwen-9B SFT checkpoint. Following the multi-harness training approach described for MiMo-V2.6, we jointly optimize the model across four mini-harnesses and evaluate it on these harnesses and three additional held-out harnesses. Table 7.2 compares Qwen3.5-9B with MiMo-V2.6-Distill-Qwen-9B before and after multi-harness RL across three coding evaluations and seven agent harnesses. MiMo-V2.6-Distill-Qwen-9B improves on Qwen3.5-9B in all 21 dataset–harness pairs, and multi-harness RL further improves every pair. On MiMo Code Bench (mini), the additional gains over SFT range from 1.8 to 9.3 percentage points across the seven harnesses.
为了更直观地展示我们的 SFT 与强化学习方案如何逐步增强模型能力，我们呈现了一个关于网站开发的案例研究（该领域的视觉质量一目了然），在图 17 中比较了由 Qwen3.5-9B、MiMo-V2.6-Distill-Qwen-9B（SFT）和 MiMo-V2.6-Distill-Qwen-9B（RL）生成的网站。如左列所示，Qwen3.5-9B 生成的网站在视觉设计上相对朴素，布局简单，图像使用极少。值得注意的是，(c) 中的人力资源管理界面存在明显的布局问题。经过蒸馏后，MiMo-V2.6-Distill-Qwen-9B（SFT）（中列）生成的网站内容明显更丰富、配色更和谐，并且更有效地融入了图像素材，例如 (b) 中埃塞俄比亚遗产页面中的摄影元素。经过强化学习训练后（右列），模型更进一步，呈现出最精致的结果：排版精炼、首屏区域视觉冲击力强（例如 (b) 中的日落图像），页面布局更完整、结构更合理（例如 (c) 中功能齐全的人力资源仪表盘，带有侧边栏导航、日历组件和详细的统计卡片）。这些逐步的定性改进与表 7.2 中观察到的定量轨迹（\(61.7 \rightarrow 64.0 \rightarrow 72.4)\）相一致，共同证明了先进行 MiMo-V2.6 蒸馏再进行强化学习在提升生成输出的美学质量和功能完整性方面的累积收益。
To offer a more intuitive view of how our SFT and RL recipe progressively enhances model capabilities, we present a case study on web development (a domain where visual quality is immediately apparent), comparing websites generated by Qwen3.5-9B, MiMo-V2.6-Distill-Qwen-9B (SFT), and MiMo-V2.6-Distill-Qwen-9B (RL) in Figure 17. As shown in the left column, websites produced by Qwen3.5-9B appear relatively plain in visual design, with simplistic layouts and minimal use of imagery. Notably, the HR management interface in (c) suffers from clear layout issues. After distillation, MiMo-V2.6-Distill-Qwen-9B (SFT) (middle column) generates websites with noticeably richer content, more harmonious color palettes, and more effective incorporation of image assets, as exemplified by the photographic elements in the Ethiopia heritage page in (b). Following RL training (right column), the model takes a further step, delivering the most polished results with refined typography, visually striking hero sections (e.g., the sunset imagery in (b)), and more complete, well-structured page layouts (e.g., the fully featured HR dashboard in (c) with sidebar navigation, a calendar widget, and detailed statistics cards). These progressive qualitative improvements align with the quantitative trajectory (\(61.7 \rightarrow 64.0 \rightarrow 72.4)\) observed in Table 7.2, jointly demonstrating the cumulative benefits of distillation from MiMo-V2.6 followed by RL in enhancing both the aesthetic quality and functional completeness of generated outputs.
本报告介绍了 MiMo-V2.6 系列，以及一条通过大规模智能体式强化学习推进基础模型的实践路径。在全能型基础模型与以智能体为中心的中期训练之上，我们从三个维度扩展强化学习：训练批大小与吞吐量、环境与智能体 harness 的多样性与复杂度，以及用于分组式智能体评分的算力。这些进展由异步训练、混合任务 rollout 基础设施、训练–推理一致性机制，以及针对训练漂移与奖励攻击的防护措施共同支撑。它们共同使得在长程交互中进行持续优化成为可能，同时提升解的质量与 token 效率。在公开与内部基准上的评估证明了该方法的有效性，MiMo-V2.6 在广泛的智能体式任务上取得了与前沿模型相竞争的性能。
This report presents the MiMo-V2.6 series and a practical approach to advancing foundation models through large-scale agentic reinforcement learning. Building on an omni-capable foundation and agent-centric mid-training, we scale RL along three dimensions: training batch size and throughput, the diversity and complexity of environments and agent harnesses, and the compute devoted to groupwise agentic grading. These advances are supported by asynchronous training, mixed-task rollout infrastructure, training–inference consistency mechanisms, and safeguards against training drift and reward hacking. Together, they enable sustained optimization over long-horizon interactions while improving solution quality and token efficiency. Evaluations across public and internal benchmarks demonstrate the effectiveness of this approach, with MiMo-V2.6 achieving competitive performance against frontier models across a broad range of agentic tasks.
为使这一方向惠及研究社区，我们发布了 MiMo-V2.6-Distill-Qwen-9B，以及精选的任务环境、验证器、端到端强化学习框架和可组合的 mini-harness。跨领域与跨 harness 的一致强化学习增益，凸显了这些资源作为进一步实验的共享基础的价值。我们的工作朝着模型自我改进迈出了实践性的一步，强调了广泛探索、信息丰富的反馈与可扩展训练系统三者的共同重要性。我们希望这些开放资源能够支持可复现研究，并推动朝着能力日益增强的通用智能体不断积累进展。
To make this direction accessible to the research community, we release MiMo-V2.6-Distill-Qwen-9B alongside curated task environments, verifiers, an end-to-end RL framework, and composable mini-harnesses. Consistent RL gains across domains and harnesses highlight the value of these resources as a shared foundation for further experimentation. Our work takes a practical step toward model self-improvement, emphasizing the joint importance of broad exploration, informative feedback, and scalable training systems. We hope these open resources support reproducible research and cumulative progress toward increasingly capable, general-purpose agents.