Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI
打开互动全文版(逐段中英对照 + 图/公式 + 针对本篇提问)→我们推出 Nemotron 3.5 Super 与 Nemotron 3.5 Lightning——面向推理、长上下文理解与智能体工作负载设计的开放、高效混合专家 Hybrid Mamba-Transformer 模型。Super 是 120B 参数、12B 激活的多模态模型,Lightning 则是更小的 30B 参数、3B 激活文本模型;两者均面向高效部署而设计。两个模型都结合了大规模蒸馏、长上下文训练与原生投机解码,Super 还进一步引入 Indic 继续预训练与智能体软件工程中期训练。我们构建了覆盖监督微调与强化学习(RLVR 与 RLHF)的完整后训练流程,Super 还辅以专门的教师模型训练与多教师同策略蒸馏(MOPD)。这使 Super 从纯文本基座模型转变为能够对文本、图像、视频及交互式工具环境进行推理的多模态智能体。我们进一步通过并行智能体执行与自验证探索测试时扩展。在训练与部署全过程中,我们联合优化能力与效率。Indic 继续预训练使印度语言基准最高提升 9.82 个百分点,智能体中期训练则增强了软件工程能力及跨智能体框架的鲁棒性。后训练后的 Super 模型在 SWE-bench Pro 上达到 51.78%,尽管激活参数约少 4.6 倍,仍超越 Nemotron 3 Ultra。后训练 MTP 增强在概率性草稿采样下将投机解码接受长度最高提升 21.8%,而 NVFP4 量化相对 BF16 实现 99.62% 的中位精度恢复。推理优化使 GB200 上每 GPU 输出吞吐最高达 4,856 tokens/s,同时保持每用户至少 50 tokens/s。
We introduce Nemotron 3.5 Super and Nemotron 3.5 Lightning, open and efficient Mixture-of-Experts Hybrid Mamba-Transformer models designed for reasoning, long-context understanding, and agentic workloads. Super is a 120B-parameter, 12B-active multimodal model, while Lightning is a smaller 30B-parameter, 3B-active text model; both designed for efficient deployment. Both models combine large-scale distillation, long-context training, and native speculative decoding, while Super further incorporates Indic continued pretraining and agentic software-engineering mid-training. We develop a comprehensive post-training pipeline spanning supervised fine-tuning and reinforcement learning (RLVR and RLHF), complemented in Super by specialized teacher training and multi- teacher on-policy distillation (MOPD). This transforms Super from a text-only base model into a multimodal agent capable of reasoning over text, images, video, and interactive tool environments. We further explore test-time scaling through parallel agent execution and self-verification. Across training and deployment, we jointly optimize capability and efficiency. Indic continued pretraining improves Indian-language benchmarks by up to 9.82 percentage points, while agentic mid-training strengthens software-engineering capabilities and robustness across agent harnesses. The post-trained Super model achieves 51.78% on SWE-bench Pro, surpassing Nemotron 3 Ultra despite using approximately 4.6x fewer active parameters. Post-training MTP boosting improves speculative- decoding acceptance length by up to 21.8% under probabilistic draft sampling, while NVFP4 quantization achieves 99.62% median accuracy recovery relative to BF16. Inference optimizations enable up to 4,856 output tokens/s/GPU on GB200 while sustaining at least 50 tokens/s/user.
我们提出 Nemotron 3.5 Super 与 Nemotron 3.5 Lightning,这是面向推理、长上下文理解与智能体式工作负载设计的开放、高效混合专家(Mixture-of-Experts)Mamba-Transformer 混合模型。Super 是一个 120B 参数、12B 激活参数的多模态模型,而 Lightning 是更小的 30B 参数、3B 激活参数的文本模型;两者均面向高效部署而设计。两个模型都结合了大规模蒸馏、长上下文训练与原生投机解码,而 Super 进一步引入了 Indic 继续预训练与智能体式软件工程中期训练。
We introduce Nemotron 3.5 Super and Nemotron 3.5 Lightning, open and efficient Mixture-of-Experts Hybrid Mamba-Transformer models designed for reasoning, long-context understanding, and agentic workloads. Super is a 120B-parameter, 12B-active multimodal model, while Lightning is a smaller 30B-parameter, 3B-active text model; both are designed for efficient deployment. Both models combine large-scale distillation, long-context training, and native speculative decoding, while Super further incorporates Indic continued pretraining and agentic software-engineering mid-training.
我们构建了一套完整的后训练流水线,涵盖监督微调与强化学习(RLVR 与 RLHF),并在 Super 中辅以专门的教师训练与多教师同策略蒸馏(MOPD)。这将 Super 从纯文本基座模型转变为能够对文本、图像、视频及交互式工具环境进行推理的多模态智能体。我们进一步通过并行智能体执行与自验证探索测试时扩展(test-time scaling)。
We develop a comprehensive post-training pipeline spanning supervised fine-tuning and reinforcement learning (RLVR and RLHF), complemented in Super by specialized teacher training and multi-teacher on-policy distillation (MOPD). This transforms Super from a text-only base model into a multimodal agent capable of reasoning over text, images, video, and interactive tool environments. We further explore test-time scaling through parallel agent execution and self-verification.
在训练与部署全过程中,我们联合优化能力与效率。Indic 继续预训练将印度语言基准提升最多 9.82 个百分点,而智能体式中期训练增强了软件工程能力以及跨智能体框架(agent harness)的鲁棒性。后训练后的 Super 模型在 SWE-bench Pro 上达到 51.78%,尽管其激活参数约少 4.6\(\times\),仍超越了 Nemotron 3 Ultra。后训练 MTP 增强在概率性草稿采样下将投机解码的接受长度最多提升 21.8%,而 NVFP4 量化相对于 BF16 实现了 99.62% 的中位精度恢复。推理优化在 GB200 上实现最高 4,856 输出 tokens/s/GPU,同时保持至少 50 tokens/s/user。在所评估的交互式智能体式工作负载下,NVFP4 达到 BF16 吞吐量的 2.43\(\times\),而 Nemotron 3.5 Super 的吞吐量分别是 Qwen3.5 与 Qwen3.8 Flash-Next 的 2.34\(\times\) 与 1.90\(\times\)。我们发布基座、BF16 后训练及 NVFP4 量化模型检查点,以及独立的并行投机草稿器(LiLiCorr、DFlash 与 DSpark),并附有模型卡、预训练与后训练数据集、cookbook 及训练配方。
Across training and deployment, we jointly optimize capability and efficiency. Indic continued pretraining improves Indian-language benchmarks by up to 9.82 percentage points, while agentic mid-training strengthens software-engineering capabilities and robustness across agent harnesses. The post-trained Super model achieves 51.78% on SWE-bench Pro, surpassing Nemotron 3 Ultra despite using approximately 4.6\(\times\) fewer active parameters. Post-training MTP boosting improves speculative-decoding acceptance length by up to 21.8% under probabilistic draft sampling, while NVFP4 quantization achieves 99.62% median accuracy recovery relative to BF16. Inference optimizations enable up to 4,856 output tokens/s/GPU on GB200 while sustaining at least 50 tokens/s/user. Under the evaluated interactive agentic workload, NVFP4 achieves 2.43\(\times\) the throughput of BF16, while Nemotron 3.5 Super delivers 2.34\(\times\) and 1.90\(\times\) the throughput of Qwen3.5 and Qwen3.8 Flash-Next, respectively. We release base, BF16 post-trained, and NVFP4-quantized model checkpoints, alongside standalone parallel speculative drafters (LiLiCorr, DFlash, and DSpark), with accompanying model cards, pretraining and post-training datasets, cookbooks, and training recipes.
我们提出了 Nemotron 3.5 Super 和 Nemotron 3.5 Lightning,这是高效的混合专家(Mixture-of-Experts)Hybrid Mamba-Transformer 模型,专为推理、多模态理解和长时运行的智能体式工作负载而设计。Nemotron 3.5 Super 是一个总参数 1200 亿、激活参数 120 亿的模型,从 Nemotron 3 Super 初始化而来;而 Lightning 3.5 是一个总参数 300 亿、激活参数 30 亿的模型,从 Nano 3 Base 初始化而来。随着 LLM 应用日益转向能够编写代码、与工具交互、在大型上下文上进行推理并在多模态环境中运行的长时程智能体,模型能力和推理效率都变得越来越重要。Nemotron 3.5 通过高效的混合架构、带有 Multi-Token Prediction(MTP)的原生推测解码、大规模蒸馏、专门的持续预训练和中期训练、面向智能体的后训练以及 NVFP4 量化来针对这一场景。
We present Nemotron 3.5 Super and Nemotron 3.5 Lightning, efficient Mixture-of-Experts Hybrid Mamba-Transformer models designed for reasoning, multimodal understanding, and long-running agentic workloads. Nemotron 3.5 Super is a 120 billion total and 12 billion active parameter model initialized from Nemotron 3 Super, while Lightning 3.5 is a 30 billion total and 3 billion active parameter model initialized from Nano 3 Base. As LLM applications increasingly move toward long-horizon agents that write code, interact with tools, reason over large contexts, and operate across multimodal environments, both model capability and inference efficiency become increasingly important. Nemotron 3.5 targets this regime through efficient hybrid architectures, native speculative decoding with Multi-Token Prediction (MTP), large-scale distillation, specialized continued pretraining and mid-training, agent-focused post-training, and NVFP4 quantization.
我们首先调整 MTP 架构以实现高效的递归推测解码。Nemotron 3.5 Super 将两个非共享的 MTP 头整合为一个递归共享头,而 Lightning 通过架构感知的投影融合和剪枝从 Nemotron 3 Super Base 初始化其 MTP 头。两个模型都在主干网络冻结的情况下进行了专门的 1000 亿 token 的 MTP 预热。对于 Super,我们还引入了隐藏状态混合,以缓解递归起草过程中训练与推理条件不匹配的问题。随后,我们使用下一个更大的 Nemotron 模型作为教师,对两个模型进行了 1 万亿 token 的蒸馏:Super 使用 Ultra 3 Base,Lightning 使用 Super 3 Base。蒸馏之后,Lightning 进行长上下文训练,而 Super 则继续进行额外的持续预训练和智能体中期训练阶段。
We begin by adapting the MTP architecture for efficient recursive speculative decoding. Nemotron 3.5 Super consolidates two non-shared MTP heads into a single recursively shared head, while Lightning initializes its MTP head from Nemotron 3 Super Base through architecture-aware projection fusion and pruning. Both models undergo a dedicated 100 billion token MTP warm-up with the backbone frozen. For Super, we additionally introduce hidden-state mixing to mitigate the train–inference conditioning mismatch during recursive drafting. We subsequently distill both models for 1 trillion tokens using the next-larger Nemotron form factor as teacher: Ultra 3 Base for Super and Super 3 Base for Lightning. Following distillation, Lightning undergoes long-context training, while Super proceeds through additional continued-pretraining and agentic mid-training stages.
我们通过有针对性的持续预训练和中期训练阶段进一步改进 Nemotron 3.5 Super。首先,我们进行了 1000 亿 token 的印度语系持续预训练,其中包含 100 亿 token 的合成和自然印度语数据,以及来自先前训练混合的 900 亿 token。为了支持这一阶段,我们使用 NeMo Data Designer 在 22 种印度语言和 34 种语言-文字配置上生成了约 2.83 万亿合成输出 token。该阶段改善了所有五个评估的印度语基准,提升高达 9.82 个百分点。然后,我们使用 350 亿 token 的持续预训练将模型扩展到 100 万 token 的上下文长度,随后在 256K 上下文下进行了 654 亿 token 的智能体软件工程中期训练。中期训练混合结合了完整仓库、问题-拉取请求对、提交历史、构建和 CI 日志以及智能体轨迹。使用相同实验性 SFT 的受控评估显示,长时程软件工程和跨智能体框架的鲁棒性有所提高,包括 DeepSWE pass@1 从 6.4% 提高到 10.4%,Codex 框架下的 SWE-bench Verified 从 48.4% 提高到 61.8%。为了恢复受 256K 中期训练影响的长上下文能力,我们进行了额外的 90 亿 token 恢复阶段,将 RULER-1M 性能从 33.5 恢复到 81.0。
We further improve Nemotron 3.5 Super through targeted continued-pretraining and mid-training stages. First, we perform 100 billion tokens of Indic continued pretraining, incorporating 10 billion tokens of synthetic and natural Indic data alongside 90 billion tokens from the preceding training mixture. To support this stage, we use NeMo Data Designer to generate approximately 2.83 trillion synthetic output tokens across 22 Indian languages and 34 language-script configurations. This stage improves all five evaluated Indic benchmarks, with gains of up to 9.82 percentage points. We then extend the model to a 1M-token context length using 35 billion tokens of continued pretraining, followed by 65.4 billion tokens of agentic software-engineering mid-training at 256K context. The mid-training mixture combines whole repositories, issue–pull-request pairs, commit histories, build and CI logs, and agent trajectories. Controlled evaluations using identical experimental SFT show improvements in long-horizon software engineering and robustness across agent harnesses, including DeepSWE pass@1 increasing from 6.4% to 10.4% and SWE-bench Verified under the Codex harness improving from 48.4% to 61.8%. To recover long-context capabilities affected by 256K mid-training, we perform an additional 9 billion token recovery stage, restoring RULER-1M performance from 33.5 to 81.0.
我们通过一个全面的流程对 Nemotron 3.5 Super 进行后训练,包括监督微调(SFT)、基于可验证奖励的强化学习(RLVR)、基于人类反馈的强化学习(RLHF)、专门的教师训练和多教师同策略蒸馏(MOPD)。我们的两阶段 SFT 流程将纯文本基础模型转变为视觉语言模型,同时联合开发推理、聊天、安全、长上下文和智能体能力。强化学习训练涵盖多样化的环境,包括奥林匹克级别的竞争性编程和需要 SEC 文件搜索、文档解析、证据检索和工具使用的金融智能体任务。我们进一步训练了用于视频推理和跟踪、视觉工具使用和对齐的专门教师,然后通过 MOPD 整合它们的能力。Nemotron 3.5 Lightning 遵循更轻量的后训练流程,包括单阶段纯文本 SFT 和 RLVR。所得的 Super 模型在 SWE-bench Pro 上达到 51.78%,尽管使用的激活参数约少 4.6 倍,仍超过了 Nemotron 3 Ultra。除了模型训练,我们还通过并行智能体执行和自我验证探索测试时扩展,将 Terminal-Bench 2.1 从 62.7% 提高到 71.9%,DeepSWE 从 24.6% 提高到 34.5%。
We post-train Nemotron 3.5 Super through a comprehensive pipeline spanning Supervised Fine Tuning (SFT), Reinforcement Learning from Verifiable Rewards (RLVR), Reinforcement Learning from Human Feedback (RLHF), specialized teacher training, and Multi-teacher On-Policy Distillation (MOPD). Our two-stage SFT pipeline transforms the text-only base model into a vision-language model while jointly developing reasoning, chat, safety, long-context, and agentic capabilities. RL training spans diverse environments, including Olympiad-level competitive programming and finance-agent tasks requiring SEC filing search, document parsing, evidence retrieval, and tool use. We further train specialized teachers for video reasoning and tracking, vision tool use, and alignment, before consolidating their capabilities through MOPD. Nemotron 3.5 Lightning follows a lighter post-training pipeline consisting of single-stage text-only SFT and RLVR. The resulting Super model achieves 51.78% on SWE-bench Pro, exceeding Nemotron 3 Ultra despite using approximately 4.6 \(\times\) fewer active parameters. Beyond model training, we explore test-time scaling through parallel agent execution and self-verification, improving Terminal-Bench 2.1 from 62.7% to 71.9% and DeepSWE from 24.6% to 34.5%.
同时,我们优化 Nemotron 3.5 以实现高效推理和部署。后训练 MTP 提升在 SPEED-Bench 上,在贪婪解码下将平均推测解码接受长度从 4.287 提高到 4.718,在概率性起草采样下从 3.787 提高到 4.612。我们开发了 NVFP4 后训练量化和混合精度方案,涵盖权重、缓存和 MTP 起草器,并辅以量化感知蒸馏,在发布的量化 Super 检查点上,在 36 个基准上实现了相对于 BF16 的 99.62% 中位精度恢复。我们还优化了混合状态缓存和 GPU 内核,以实现高效的长上下文和智能体服务,并开发了用于低并发、内存受限推理的独立并行推测起草器。在单个 GB200 GPU 上,Nemotron 3.5 Super 在 20 轮智能体代理工作负载上实现了高达 4,856 输出 token/s/GPU 的吞吐量,同时保持至少 50 token/s/用户。在评估的交互式服务条件下,NVFP4 的吞吐量是 BF16 的 2.43 倍,而 Super 的吞吐量分别是 Qwen3.5 122B-A10B 和 Qwen3.8 Flash-Next 的 2.34 倍和 1.90 倍。我们发布了基础、BF16 后训练和 NVFP4 量化的模型检查点,以及独立的并行推测起草器(Super 的 LiLiCorr,Lightning 的 DFlash 和 DSpark),并附有模型卡、预训练和后训练数据集、烹饪书和训练配方。
In parallel, we optimize Nemotron 3.5 for efficient inference and deployment. Post-training MTP boosting improves average speculative-decoding acceptance length from 4.287 to 4.718 under greedy decoding and from 3.787 to 4.612 under probabilistic draft sampling on SPEED-Bench. We develop NVFP4 post-training quantization and mixed-precision recipes across weights, caches, and the MTP drafter, complemented by quantization-aware distillation, achieving 99.62% median accuracy recovery relative to BF16 across 36 benchmarks for the released quantized Super checkpoint. We additionally optimize hybrid-state caching and GPU kernels for efficient long-context and agentic serving, and develop standalone parallel speculative drafters for low-concurrency, memory-bound inference. On a single GB200 GPU, Nemotron 3.5 Super achieves up to 4,856 output tokens/s/GPU on a 20-turn agentic proxy workload while sustaining at least 50 tokens/s/user. Under the evaluated interactive serving conditions, NVFP4 delivers 2.43 \(\times\) the throughput of BF16, while Super achieves 2.34 \(\times\) and 1.90 \(\times\) the throughput of Qwen3.5 122B-A10B and Qwen3.8 Flash-Next, respectively. We release base, BF16 post-trained, and NVFP4-quantized model checkpoints, alongside standalone parallel speculative drafters (LiLiCorr for Super, and DFlash and DSpark for Lightning), with accompanying model cards, pretraining and post-training datasets, cookbooks, and training recipes.
Nemotron 3.5 Super VL 120B-A12B Base BF16:基础模型
Nemotron 3.5 Super VL 120B-A12B Base BF16: base model
Nemotron 3.5 Super VL 120B-A12B BF16:后训练模型
Nemotron 3.5 Super VL 120B-A12B BF16: post-trained model
Nemotron 3.5 Super VL 120B-A12B NVFP4:经后训练并采用 NVFP4 量化的模型
Nemotron 3.5 Super VL 120B-A12B NVFP4: post-trained and NVFP4-quantized model
Nemotron 3.5 Super VL 120B-A12B NVFP4 LiLiCorr:LiliCorr
Nemotron 3.5 Super VL 120B-A12B NVFP4 LiLiCorr: LiliCorr
Nemotron 3.5 Lightning 30B-A3B Base BF16:基础模型
Nemotron 3.5 Lightning 30B-A3B Base BF16: base model
Nemotron 3.5 Lightning 30B-A3B BF16:后训练模型
Nemotron 3.5 Lightning 30B-A3B BF16: post-trained model
Nemotron 3.5 Lightning 30B-A3B NVFP4:后训练并经 NVFP4 量化的模型
Nemotron 3.5 Lightning 30B-A3B NVFP4: post-trained and NVFP4-quantized model
Nemotron 3.5 Lightning 30B-A3B NVFP4 DFlash:DFlash
Nemotron 3.5 Lightning 30B-A3B NVFP4 DFlash: DFlash
Nemotron 3.5 Lightning 30B-A3B NVFP4 DSpark:DSpark
Nemotron 3.5 Lightning 30B-A3B NVFP4 DSpark: DSpark
本报告其余部分组织为四节:预训练(§2)、后训练(§3)、量化(§4)与推理(§5)。
We organize the remainder of the report into four sections: Pretraining (§2), Post-training (§3), Quantization (§4), and Inference (§5).
本节介绍用于开发 Nemotron 3.5 Super 与 Lightning 3.5 的预训练与中期训练方案。我们首先为递归投机解码适配并预热多词预测(Multi-Token Prediction, MTP)头,随后以次一级规模的 Nemotron 3 模型作为教师,对两个模型各蒸馏 1 万亿 token。蒸馏之后,两个模型均进行长上下文训练。对于 Super,我们还额外施加 Indic 继续预训练,在 256K 上下文窗口下进行智能体式软件工程中期训练,并以一个简短的 1M 上下文恢复阶段收尾。我们还报告了所得基座模型的能力,以及这些阶段对多语言、长上下文与智能体式性能的影响。
In this section, we describe the pretraining and mid-training recipe used to develop Nemotron 3.5 Super and Lightning 3.5. We first adapt and warm up the Multi-Token Prediction (MTP) head for recursive speculative decoding, then distill both models for 1 trillion tokens using the next-larger Nemotron 3 model as teacher. Following distillation, both models undergo long-context training. For Super, we additionally apply Indic continued pretraining, perform agentic software-engineering mid-training at 256K context, and finish with a short 1M-context recovery stage. We also report the resulting base-model capabilities and the effects of these stages on multilingual, long-context, and agentic performance.
Nemotron 3.5 Super 和 Nemotron 3.5 Lightning 通过多 Token 预测(MTP)头原生支持推测解码,采用了 中详述的递归共享 MTP 公式。MTP 头作为一个集成的内部草稿器,从骨干网络的隐藏表示中预测多个未来 Token。这使得在骨干网络的一次前向传播中,通过拒绝采样或精确 Token 匹配进行多 Token 候选验证成为可能。
Nemotron 3.5 Super and Nemotron 3.5 Lightning feature native speculative-decoding support via a Multi-Token Prediction (MTP) head, adopting the recursively shared MTP formulation detailed in . The MTP head functions as an integrated internal drafter that predicts multiple future tokens from the backbone's hidden representations. This enables multi-token candidate verification in a single forward pass over the backbone using rejection sampling or exact token matching.
为了建立 MTP 草稿头,我们根据 Super 和 Lightning 变体的架构起源,采用了针对特定模型的初始化策略:
To establish the MTP drafting head, we employ model-specific initialization strategies tailored to the architectural origins of the Super and Lightning variants:
Nemotron 3.5 Super:我们直接从 Nemotron 3 Super Base 检查点(在长上下文扩展之前)初始化 MTP 权重,该检查点最初包含两个不同的 MTP 头。为了将多 Token 草稿整合到一个递归共享的单一头中,同时保留底层表示空间,我们仅从第一个 MTP 头初始化 MTP 参数,并丢弃第二个。
Nemotron 3.5 Super: We initialize the MTP weights directly from the Nemotron 3 Super Base checkpoint (prior to long-context extension), which originally contained two distinct MTP heads. To consolidate multi-token drafting into a single, recursively shared head while preserving the underlying representation space, we initialize the MTP parameters exclusively from the first MTP head and discard the second.
Nemotron 3.5 Lightning:我们从 Nemotron 3 Super Base 的 MTP 头初始化 Lightning 的 MTP 头。我们首先将潜在 MoE 投影融合到专家的上投影和下投影矩阵中,然后将隐藏大小从 4096 修剪到 2688,MoE 中间大小从 2688 修剪到 1856,路由专家数量从 512 修剪到 128,以匹配 Lightning。
Nemotron 3.5 Lightning: We initialize Lightning's MTP head from that of Nemotron 3 Super Base. We first fuse the latent MoE projections into the experts' up- and down-projection matrices, then prune the hidden size from 4096 to 2688, MoE intermediate size from 2688 to 1856, and number of routed experts from 512 to 128 to match Lightning.
初始化后,我们进行了专门的 MTP 预热阶段,使用 \(100\,\text{B}\) 个 Token,冻结骨干网络参数,仅优化 MTP 头。对于 Nemotron 3.5 Lightning,我们应用了标准的 2 步递归 MTP 训练。然而,这种朴素的训练导致了明显的训练-推理条件不匹配。在标准训练中,递归步骤 \(k\) 以步骤 \(k-1\) 生成的真值隐藏状态的完整移位序列为条件。相比之下,在推理时的自回归草稿中,MTP 头以骨干网络隐藏状态和由先前 MTP 步骤生成的合成状态的日益异质的混合为条件。这种差异随着草稿深度的增加而加剧,降低了后续位置的 Token 接受率。
Following initialization, we conduct a dedicated MTP warmup phase for \(100\,\text{B}\) tokens, freezing the backbone parameters and optimizing only the MTP head. For Nemotron 3.5 Lightning, we applied the standard 2-step recursive MTP training. However, this naive training induces a pronounced train–inference conditioning mismatch. During standard training, recursive step \(k\) conditions on the full shifted sequence of ground-truth hidden states generated at step \(k-1\). In contrast, during autoregressive drafting at inference time, the MTP head conditions on an increasingly heterogeneous mixture of backbone hidden states and synthetic states generated by preceding MTP steps. This discrepancy compounds with increasing draft depth, degrading token acceptance rates at later positions.
为弥合 Nemotron 3.5 Super 中的这一分布差距,我们采用了一种类似于 [引用] 的隐藏状态混合策略。在训练步 \(k\) 时,输入隐藏状态从先前 MTP 各步 \(1, \ldots, k-1\) 所产生的状态序列中采样,而非仅依赖第 \(k-1\) 步。在预热阶段让 MTP 头接触异构的隐藏状态混合,可增强其在更深草稿步上的鲁棒性,从而在不需测试时训练(TTT)所要求的高计算量多步模拟的情况下,获得显著的接受率提升。
To bridge this distribution gap in Nemotron 3.5 Super, we adopt a hidden-state-mixing strategy similar to [citation]. At training step \(k\), input hidden states are sampled from the sequence of states produced by earlier MTP steps \(1, \ldots, k-1\), rather than relying exclusively on step \(k-1\). Exposing the MTP head to heterogeneous hidden-state mixtures during warmup enhances robustness across deeper draft steps, achieving significant acceptance rate gains without the computationally intensive multi-step simulation required by test-time training (TTT).
我们以 Super 3 Base 初始化 Super 3.5 Base,以 Nano 3 Base 初始化 Lightning 3.5 Base,然后对每个模型各蒸馏 1 万亿个 token。
We initialized Super 3.5 Base from Super 3 Base and Lightning 3.5 Base from Nano 3 Base, then distilled each model for 1 trillion tokens.
对于 Lightning 3.5 Base,我们在 1 万亿个 Ultra 3 Phase 2 数据 token 上使用了 Super 3 Base 的 top-128 logits。我们使用标准的前向 KL 散度损失,并额外加入一个 sink token 以捕获 top-128 之外的剩余概率质量。为了给稀有 token 提供训练信号,我们将 KL 损失与交叉熵损失相结合,权重分别设为 0.9 和 0.1。为限制早期偏离初始化,我们以 Nano 3 Base 的最小学习率(LR)(\(10^{-5})\)开始蒸馏,并在 2000 亿个 token 上缓慢提升至 \(10^{-4})\,即 Nano 3 Base 峰值 LR 的 10%。随后在剩余的 8000 亿个 token 上,我们将 LR 线性衰减回 \(10^{-5})\。除新增的 MTP 层外,所有其他超参数和设置均与 Nano 3 Base 一致。我们使用的 MTP 损失缩放因子为 0.1。
For Lightning 3.5 Base, we used top-128 logits from Super 3 Base on 1 trillion tokens of the Ultra 3 Phase 2 data. We used standard forward KL divergence loss along with an extra sink token to capture the residual probability mass outside top-128. To provide training signal for rare tokens, we combined the KL loss with cross-entropy loss, assigning them weights of 0.9 and 0.1, respectively. To limit early drift from the initialization, we began distillation with the minimum Learning Rate (LR) of Nano 3 Base (\(10^{-5})\) and ramped it up slowly to \(10^{-4})\, 10% of the Nano 3 Base's peak LR, over 200 billion tokens. We then decayed the LR linearly back to \(10^{-5})\ over the remaining 800 billion tokens. All other hyperparameters and settings match Nano 3 Base, except for the added MTP layers. We used an MTP loss scaling factor of 0.1.
对于 Super 3.5 Base,我们采用与 Lightning 3.5 Base 相同的蒸馏配方,但以 Ultra 3 Base 作为教师模型,并使用 Skew Reverse KL 散度作为蒸馏损失函数,其中 \(\alpha=0.1\)\。此外,我们使用了更小的 MoE 序列负载均衡损失系数 \(10^{-5})\,而 Lightning 3.5 为 \(10^{-4})\。我们以与 Lightning 3.5 Base 相同的方式选择学习率,但以 Super 3 Base 的学习率为锚点,使用 \(4.5\times 10^{-6})\ 作为最小值,\(4.5\times 10^{-5})\ 作为最大值。
For Super 3.5 Base, we used the same distillation recipe as Lightning 3.5 Base but with Ultra 3 Base as the teacher and Skew Reverse KL divergence as the distillation loss function with \(\alpha=0.1\)\. Additionally, we used a smaller MoE sequence load balancing loss coefficient of \(10^{-5})\, as compared to \(10^{-4})\ for Lightning 3.5. We chose learning rates the same way as Lightning 3.5 Base, but with Super 3 Base LRs as the anchor, using \(4.5\times 10^{-6})\ as the minimum and \(4.5\times 10^{-5})\ as the maximum.
我们为 Nemotron 3.5 Super 增加了一个 Indic 继续预训练阶段,该阶段将从英文文档生成的合成文本与自然 Indic 语料库相结合,以增强印度语言能力。我们的十二语言合成数据流水线使用 Gemma-4-31B-it 进行翻译。我们描述了数据和生成流水线,然后展示了 Super 的训练混合和结果,随后是单独的 Lightning 3.5 混合消融实验。
We augment Nemotron 3.5 Super with an Indic continued-pretraining stage that combines synthetic text generated from English documents with natural Indic corpora to strengthen Indian-language capabilities. Our twelve-language synthetic-data pipeline uses Gemma-4-31B-it for translation. We describe the data and generation pipeline, then present Super's training mixture and results, followed by separate Lightning 3.5 mixture ablations.
自然 Indic 文本在许多语言中很稀缺,其覆盖范围因语言、文字和主题而异。我们通过将精选的英文文档翻译成 Indic 语言来弥补这些差距。Nemotron-CC 提供了大部分英文源文档。来源涵盖一般网络内容、教育、数学、STEM 和问答。保留源文档使我们能够检查生成的文本是否保持了其意义和结构。
Natural Indic text is scarce in many languages, and its coverage varies across languages, scripts, and subjects. We address these gaps by translating curated English documents into Indic languages. Nemotron-CC provides most of the English source documents. The sources cover general web content, education, mathematics, STEM, and question-answering. Keeping the source documents lets us check whether the generated text preserves their meaning and structure.
我们生成了涵盖 22 种印度语言的数据:十二种优先语言的原生文字和罗马化文本,以及十种额外低资源语言的原生文字文本。Nemotron-Pretraining-Indic-v1 语料库包含总共约 2.83 万亿个生成的 Indic 文本 token,涵盖 34 种语言-文字配置,使用 Gemma-4 分词器计数(表 1)。十二种优先语言贡献了约 2.50 万亿 token,包括 2246 亿罗马化 token;额外的十种语言贡献了约 3270 亿原生文字 token。这种规模和覆盖范围为跨语言和主题提供了大型、可重用的训练文本池,包括自然文本有限的语言。模型构建者可以利用这个池来研究他们自己模型的语言混合、文字选择和训练预算。
We generated data across 22 Indian languages: native-script and romanized text for twelve priority languages, and native-script text for ten additional lower-resource languages. The Nemotron-Pretraining-Indic-v1 corpus contains a total of approximately 2.83 trillion tokens of generated Indic text across 34 language-script configurations, counted using a Gemma-4 tokenizer (Table 1). The twelve priority languages contribute approximately 2.50 trillion tokens, including 224.6 billion romanized tokens; the additional ten languages contribute approximately 327 billion native-script tokens. This scale and coverage provide a large, reusable pool of training text across languages and subjects, including languages where natural text is limited. Model builders can draw on this pool to study language mixtures, script choices, and training budgets for their own models.
我们创建的数据远多于 Super 的 Indic CPT 阶段所使用的数据,并将公开发布更广泛的语料库。其他团队可以直接使用数据,检查其生成方式,并将生成扩展到自己的需求。
We created substantially more data than was used in Super's Indic CPT stage and will release the broader corpus publicly. Other teams can use the data directly, inspect how it was produced, and extend generation to their own needs.
我们使用 NeMo Data Designer 为十二种优先语言实现了四阶段流水线(图 2)。Gemma-4-31B-it 基于翻译模型比较被选中,执行翻译、质量评估和重写。我们选择少于 4,096 个 Gemma-4 token 的源文档,并使用旨在保留事实、数学表达式、代码和格式的提示,完整翻译每个文档而不拆分。然后模型对照源文档检查每个翻译,并分配 0 到 100 的质量分数。我们保留得分至少为 75 的翻译。得分较低的翻译使用源文本和识别出的错误进行重写,然后再次检查,仅当通过时才保留。
We implement the four-stage pipeline for the twelve priority languages (Figure 2) using NeMo Data Designer. Gemma-4-31B-it, selected based on translation-model comparisons, performs translation, quality assessment, and rewriting. We select source documents with fewer than 4,096 Gemma-4 tokens and translate each document in full, without splitting, using prompts designed to preserve facts, mathematical expressions, code, and formatting. The model then checks each translation against its source and assigns a quality score from 0 to 100. We keep translations scoring at least 75. Lower-scoring translations are rewritten using the source text and identified errors, then checked again and kept only if they pass.
保留的译文会保留其来源和质量元数据,以便追溯。质量分数用于指导过滤,不过用同一个模型既做翻译又做评判,可能会偏向它自己的译文。
Retained translations keep their source and quality metadata for traceability. Quality scores guide filtering, although using the same model for translation and judging may favor its own translations.
对于另外十种资源较少的语言,我们使用 Bodhan-MT 进行翻译,使用 Gemma-4-31B-it 进行质量评估,并采用针对各语言设定的接受阈值。我们仅对较长的 Santali 文档使用重叠窗口,并将 Manipuri 译文从孟加拉文转写为 Meitei Mayek 文。配套报告《Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs》提供了详细信息。
For the ten additional lower-resource languages, we use Bodhan-MT for translation and Gemma-4-31B-it for quality assessment, with language-specific acceptance thresholds. We use overlapping windows only for longer Santali documents and convert Manipuri translations from Bengali script to Meitei Mayek. The companion Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs report provides details.
对于十二种语言的流水线,我们改进了调度,使翻译和质量检查可以并发运行。推测解码、FP8 权重和 FlashAttention 将每百万英文输入 token 的翻译成本中位数从 0.835 GPU 小时降至 0.335 GPU 小时,降幅约 60%。这相当于每 GPU 小时处理的 token 数提升至 2.49 倍。该对比使用相同的十二语言基准,每种语言约 46.8M 输入 token。这一收益适用于翻译;质量检查和改写还会带来额外成本。
For the twelve-language pipeline, we improved scheduling so translation and quality checks can run concurrently. Speculative decoding, FP8 weights, and FlashAttention reduced median translation cost from 0.835 to 0.335 GPU-hours per million English input tokens, a reduction of about 60%. This is equivalent to processing 2.49 times as many tokens per GPU-hour. The comparison uses the same twelve-language benchmark, with approximately 46.8M input tokens per language. The gain applies to translation; quality checks and rewriting add further cost.
Super 的 Indic 继续预训练运行名义上有 100B token 的预算,其中 10%(10B token)分配给从我们的语料中选出的合成文本和自然 Indic 文本。其余 90%(90B token)保持先前训练混合中相对来源比例不变。这些是配置的采样分配,而非实测的各来源 token 总量或语料中唯一 token 的计数。
Super's Indic continued-pretraining run has a nominal 100B-token budget, with 10% (10B tokens) allocated to selected synthetic text from our corpus and natural Indic text. The remaining 90% (90B tokens) retains the preceding training mixture's relative source proportions. These are configured sampling allocations, not measured per-source token totals or counts of unique tokens in the corpus.
我们将 10% 的 Indic 占比选为该预算内的一个实用分配,这一选择参考了单独的 200B token Lightning 混合消融实验(表 3)。10% 的混合已经带来了显著收益;更高占比进一步提升了结果,但增量收益更小。
We chose the 10% Indic share as a practical allocation within this budget, informed by the separate 200B-token Lightning mixture ablations (Table 3). The 10% mixture already delivered substantial gains; higher shares improved results further, but with smaller incremental gains.
我们使用 MILU 以及经人工核验的 MMLU-Pro、ARC-Challenge、ARC-Easy 和 GSM8K 印度语言译本,评估印度语言知识、推理与数学能力。
We evaluate Indian-language knowledge, reasoning, and mathematics using MILU and human-verified Indic translations of MMLU-Pro, ARC-Challenge, ARC-Easy, and GSM8K.
MILU 覆盖十种受支持语言;GSM8K 排除了 Odia,因为其评估超出了服务上下文窗口限制。翻译在十二种语言的 IGB/FLORES 上也有提升:英语到印度语言的 chrF++ 从 35.43 升至 47.47,印度语言到英语从 58.75 升至 64.30。后续的 midtraining 检查点基本保留了印度语言基准上的增益。
MILU covers ten supported languages; GSM8K excludes Odia because its evaluation exceeded the serving context limit. Translation also improves on twelve-language IGB/FLORES: English-to-Indic chrF++ rises from 35.43 to 47.47, and Indic-to-English from 58.75 to 64.30. The later midtraining checkpoint largely retains the Indic benchmark gains.
我们还在大量英语基准上评估了 Indic-CPT 模型,未在测试类别中观察到性能回退。
We also evaluated the Indic-CPT model across a broad set of English benchmarks and observed no regressions in the tested categories.
为补充这一阶段级评估,图 3 在更广泛的印度语言任务和语言质量指标上比较了所评估的 Nemotron 3 与 3.5 Super 模型。Nemotron 3.5 Super 在所示六项任务指标中的五项以及全部四个 IndiVibe 维度上得分更高。这些结果展示了已发布 Nemotron 3.5 Super 模型的整体性能,其中 Indic CPT 是其更广泛训练流程的一部分。
To complement this stage-level evaluation, Figure 3 compares the evaluated Nemotron 3 and 3.5 Super models on a broader set of Indic tasks and language-quality measures. Nemotron 3.5 Super scores higher on five of the six task metrics shown and on all four IndiVibe dimensions. These results show the overall performance of the released Nemotron 3.5 Super model, with Indic CPT as part of its broader training pipeline.
为研究印度语言数据占比如何影响性能,我们在固定 200B token 训练预算下,分别以 10%、20% 和 30% 的印度语言数据开展了独立的 Lightning 3.5 研究实验。这些实验与已发布的 Lightning 3.5 模型相互独立,后者未包含此 Indic CPT 阶段。四项基准的平均值随印度语言占比上升而提高,其中 30% 混合比例在测试设置中得分最高(表 3)。这些实验为混合比例设计提供参考,但并未确定 Super 的最优占比。详细的语言级结果、混合配置以及英语保持分析将出现在配套报告《Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs》中。
To study how the Indic share affects performance, we ran separate Lightning 3.5 research experiments with 10%, 20%, and 30% Indic data under a fixed 200B-token training budget. These experiments are separate from the released Lightning 3.5 model, which did not include this Indic CPT stage. The four-benchmark mean increased with the Indic share, with the 30% mixture scoring highest among the tested settings (Table 3). These experiments inform mixture design but do not establish an optimal share for Super. Detailed language-level results, mixture configurations, and English-retention analysis will appear in the companion Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs report.
与 Nemotron 3.5 技术报告一同,我们将发布更广泛的合成语料库、开源生成流水线,并发布一份配套技术报告。该报告将提供详细的翻译模型对比、在 Nemotron 3 Nano 上的数据源消融实验,以及 Lightning 3.5 混合实验。我们还将与 AI4Bharat 合作,发布经人工验证的印度语 LLM 评测套件。这些资源将共同帮助模型构建者训练、评估和适配面向印度语言应用的模型。
Alongside the Nemotron 3.5 technical report, we will release the broader synthetic corpus, open-source the generation pipelines, and publish a companion technical report. That report will provide detailed translation-model comparisons, data-source ablations on Nemotron 3 Nano, and Lightning 3.5 mixture experiments. In collaboration with AI4Bharat, we will also release a human-verified Indic LLM evaluation suite. Together, these resources will help model builders train, evaluate, and adapt models for Indian-language applications.
与 Nemotron 3 系列类似,我们在预训练末尾加入了一个长上下文阶段(LC-Phase)。在 LC-Phase 中,我们以 1M token 的上下文窗口进行持续预训练(CPT),使基础模型具备长上下文能力。对于 Lightning 3.5,我们使用恒定的学习率 \(1*10^{-5}\)。我们在 GB200 GPU 上采用 8 路上下文并行、8 路张量并行、8 路专家并行和 2 路流水线并行进行训练。对于 Super 3.5,我们使用恒定的学习率 \(4.5*10^{-6}\)。我们在 GB300 GPU 上采用 16 路上下文并行、8 路张量并行和 64 路专家并行进行训练。
Similar to the Nemotron 3 series, we added a long-context phase (LC-Phase) at the end of pretraining. In the LC-Phase, we performed continued pretraining (CPT) at 1M-token context length to equip the base model with long-context ability. For Lightning 3.5, we used a constant learning rate of \(1*10^{-5}\). We used 8-way context parallelism, 8-way tensor parallelism, 8-way expert parallelism, and 2-way pipeline parallelism to train on GB200 GPUs. For Super 3.5, we used a constant learning rate of \(4.5*10^{-6}\). We used 16-way context parallelism, 8-way tensor parallelism, and 64-way expert parallelism to train on GB300 GPUs.
我们使用与 Nemotron 3 Ultra 相同的数据混合。LC-Phase 阶段中,Lightning 训练了 21B token,Super 训练了 35B token。需要注意的是,对于 Super 3.5,在以 256K 上下文窗口完成中训练之后,我们观察到 RULER-1M 分数从 79.9 下降到 33.5。因此,我们在中训练之后加入了一个简短的长上下文恢复阶段,使用相同的配方训练了 9B token,并成功将 RULER 分数恢复到 81.0。
We use the same data blend as Nemotron 3 Ultra. The LC-Phase was trained for 21B tokens for Lightning and 35B tokens for Super. Note that for Super 3.5 after mid-training which was performed at a context length of 256K, we observed that the RULER-1M score dropped from 79.9 to 33.5. So we added a short long-context recovery phase after mid-training, for which we trained 9B tokens with the same recipe and successfully recovered the RULER score to 81.0.
在长上下文阶段之后,我们为智能体式软件工程增加了一个中期训练阶段。预训练在短上下文下运行,几乎看不到智能体交互,而围绕软件工程的自然数据(例如完整代码仓库、带有 issue 的 pull request、提交历史和构建日志)往往对它来说过长。SFT 则主要由合成数据构成,而过长的 SFT 阶段会削弱模型之后从强化学习中获得的收益。中期训练介于两者之间。它在长上下文下用自然软件工程数据与少量智能体轨迹的多样混合来训练基础模型,并把行为塑造留给后训练。完整方案(包括每个数据源如何构建与过滤,以及每个选择背后的消融实验)见。
After the long-context phase, we add a mid-training stage for agentic software engineering. Pretraining runs at short context and sees almost no agent interactions, and the natural data around software engineering, such as whole repositories, pull requests with their issues, commit histories, and build logs, is often too long for it. SFT in turn consists mostly of synthetic data, and very long SFT stages can reduce how much the model later gains from RL. Mid-training sits between the two. It trains the base model at long context on a diverse mixture of natural software-engineering data with a small share of agent traces, and leaves behavior shaping to post-training. The full recipe, including how each data source is built and filtered and the ablations behind each choice, is described in .
我们以 256K 的序列长度对长上下文检查点进行了 65.4B token 的中期训练。我们重置了学习率,将其线性预热至最大值 \(1\times 10^{-5}\)(约为预训练最终学习率的两倍),然后在剩余训练中保持恒定。在 256K 下训练降低了 RULER-1M 分数,因此我们在中期训练之后接了一个约 9B token 的短长上下文恢复阶段,详见上文的长上下文一节。
We mid-trained the long-context checkpoint for 65.4B tokens at a sequence length of 256K. We reset the learning rate, warmed it up linearly to a maximum of \(1\times 10^{-5}\), about twice the final pretraining learning rate, and then held it constant for the rest of the run. Training at 256K lowered the RULER-1M score, so we followed mid-training with a short long-context recovery stage of about 9B tokens, described in the Long Context section above.
为衡量中期训练的效果,我们在基础检查点和中期训练后的检查点上运行了相同的 SFT,并对两者进行比较。该 SFT 是一个约 33B token(2K 步)的实验性智能体混合数据,并非第 3 节所用的那个,但我们发现它能可靠地预示检查点在完整后训练之后的表现。表 5 展示了结果。在标准基准上最明显的提升出现在 DeepSWE——我们最难的长时程基准——其 pass@1 从 6.4 升至 10.4,BrowseComp 略有提升。在 OpenHands 下的 SWE-bench Verified、Multilingual 和 Pro 则处于运行间波动范围内,约为一个点。更大的影响体现在跨智能体 harness 的鲁棒性上。在 OpenHands(我们标准评估所用的 harness)下,基础模型本已得分良好,中期训练改变不大。在另外五个 harness 下,中期训练后的模型在每一个上都更好:SWE-bench Verified 平均提升 8.6 点,Terminal-Bench 2.1 提升 6.8 点,其中在 Codex 上提升最大——基础模型在该 harness 上最弱。
To measure the effect of mid-training, we ran identical SFT on the base checkpoint and on the mid-trained checkpoint and compared the two. This SFT is an experimental agentic blend of about 33B tokens (2K steps), not the one used in Section 3, but we have found it to be a reliable signal of how a checkpoint will do after full post-training. Table 5 shows the results. The clearest gain on the standard benchmarks is on DeepSWE, our hardest long-horizon benchmark, where pass@1 rises from 6.4 to 10.4, and BrowseComp improves slightly. SWE-bench Verified, Multilingual, and Pro under OpenHands stay within run-to-run variation, which is about one point. The larger effect is on robustness across agent harnesses. Under OpenHands, the harness used for our standard evaluations, the base model already scores well and mid-training changes little. Under five other harnesses, the mid-trained model is better on every one, by 8.6 points on average on SWE-bench Verified and 6.8 points on Terminal-Bench 2.1, with the largest gain on Codex, where the base model is weakest.
我们在涵盖通用知识、数学推理、代码生成、常识推理、阅读理解以及多语言能力的广泛基准测试套件上对基础模型进行了评估。详细的评估设置和结果如下所示。
We evaluate the base models across a broad suite of benchmarks covering general knowledge, mathematical reasoning, code generation, commonsense reasoning, reading comprehension, and multilingual capabilities. Detailed evaluation settings and results are provided below.
除非另有说明,Nemotron 3.5 Super 120B-A12B Base 和 Lightning 3.5 30B-A3B Base 的所有评估结果均通过 NeMo Evaluator SDK[^1] 和 NVIDIA 开源的 LM Evaluation Harness 容器[^2] 收集。为确保可复现性,评估配置的更多细节可在 NeMo Evaluator SDK 示例[^3] 中获取。通过 NVIDIA NeMo Evaluator SDK 打包的开源 LM Evaluation Harness 容器可在 NGC[^4] 上获取。
All evaluation results reported for Nemotron 3.5 Super 120B-A12B Base and Lightning 3.5 30B-A3B Base were collected via NeMo Evaluator SDK[^1] and NVIDIA's open-source container of LM Evaluation Harness[^2], unless otherwise stated. For reproducibility, additional details on evaluation configurations are available in the NeMo Evaluator SDK examples[^3]. The open-source LM Evaluation Harness container packaged through NVIDIA's NeMo Evaluator SDK is available on NGC[^4].
评估使用以下设置:
The evaluations use the following settings:
对于通用知识,我们评估了 MMLU、MMLU-Pro、AGIEval-En、SuperGPQA 和 GPQA Diamond。在适用的情况下,我们使用针对基准的特定提取程序报告精确匹配指标。GPQA Diamond 使用 5-shot 思维链提示进行评估。
For general knowledge, we evaluate MMLU, MMLU-Pro, AGIEval-En, SuperGPQA, and GPQA Diamond. We report exact-match metrics using benchmark-specific extraction procedures where applicable. GPQA Diamond is evaluated using 5-shot chain-of-thought prompting.
对于数学推理,我们使用 8-shot 思维链精确匹配评估 GSM8K,并使用精确匹配评估 Minerva MATH。此外,我们使用 4-shot 采样生成(每个提示 32 次生成)评估 MATH-500,并报告 pass@1。
For mathematical reasoning, we evaluate GSM8K using 8-shot chain-of-thought exact match and Minerva MATH using exact match. We additionally evaluate MATH-500 using 4-shot sampled generation with 32 generations per prompt and report pass@1.
对于代码任务,我们使用经 EvalPlus 清洗的变体来评估 HumanEval 和 MBPP。我们报告基于每个提示 32 次生成所估计的采样 pass@1。MBPP-Sanitized 使用 3-shot 提示,而 HumanEval 使用零样本提示。我们还额外报告了在 LiveCodeBench CodeGen 以及 BigCodeBench 的 complete 和 instruct 变体上的结果。
For code tasks, we evaluate HumanEval and MBPP using EvalPlus-sanitized variants. We report sampled pass@1 estimated from 32 generations per prompt. MBPP-Sanitized uses 3-shot prompting, while HumanEval uses zero-shot prompting. We additionally report results on LiveCodeBench CodeGen and the complete and instruct variants of BigCodeBench.
对于常识推理,我们使用表 6 中所示的准确率或归一化准确率指标来评估 CommonsenseQA、ARC-Challenge、HellaSwag、OpenBookQA、PIQA、TruthfulQA MC2、WinoGrande 和 SocialIQA。
For commonsense reasoning, we evaluate CommonsenseQA, ARC-Challenge, HellaSwag, OpenBookQA, PIQA, TruthfulQA MC2, WinoGrande, and SocialIQA using the accuracy or normalized-accuracy metrics shown in Table 6.
对于阅读理解,我们使用零样本准确率来评估 RACE。
For reading comprehension, we evaluate RACE using zero-shot accuracy.
对于多语言能力,我们在德语、西班牙语、法语、意大利语、日语、韩语、葡萄牙语和中文上评估 Global MMLU,并报告这八种语言的平均准确率。我们使用西班牙语、德语、法语、中文、日语和俄语的 8-shot 母语思维链提示来评估 MGSM,并报告这六种语言的平均精确匹配得分。
For multilingual capability, we evaluate Global MMLU across German, Spanish, French, Italian, Japanese, Korean, Portuguese, and Chinese, and report the average accuracy across these eight languages. We evaluate MGSM using 8-shot native chain-of-thought prompting in Spanish, German, French, Chinese, Japanese, and Russian, and report the average exact-match score across the six languages.
Nemotron 3.5 Super 120B-A12B Base 和 Lightning 3.5 30B-A3B Base 的评估结果汇总于表 6。
Evaluation results for Nemotron 3.5 Super 120B-A12B Base and Lightning 3.5 30B-A3B Base are summarized in Table 6.
Super 的后训练流程与 Nemotron 3 Ultra 中所采用的流程类似,由 SFT、RLVR、专家教师模型训练以及最后的 MOPD 组成,如图 4 所示。我们将在后续各节中详细阐述该流程的每个阶段。Lightning 则采用更轻量的流程,仅包含纯文本 SFT 和 RLVR。
Our post-training pipeline for Super is similar to that used in Nemotron 3 Ultra, consisting of SFT, RLVR, expert teacher model training, and finally MOPD as shown in Figure 4. We elaborate on each stage of the pipeline in the following sections. Lightning goes through a lighter pipeline, with text-only SFT and RLVR.
我们将 Nemotron 3.5 Super 训练为 VLM,将 Nemotron 3.5 Lightning 训练为纯文本模型。由于 Nemotron Super 3.5 的基础模型是在纯文本数据上预训练的,我们在 SFT 阶段引入了大量视觉数据,以及智能体式与推理文本数据。在对视觉投影器进行预热之后,我们采用两阶段 SFT 流程,因为同时学习视觉与文本能力颇具挑战。在第一阶段,我们在视觉、推理、对话、长上下文、安全与智能体式数据的通用混合上训练。随后我们进行一个小规模的第二阶段 SFT,调整混合数据的组成,以提升第一阶段未能充分学会的某些能力。Nemotron 3.5 Lightning 则使用纯文本 SFT 数据、以单阶段 SFT 训练。
We train Nemotron 3.5 Super to be a VLM and Nemotron 3.5 Lightning to be a text-only model. As the base model for Nemotron Super 3.5 is pre-trained on text-only data, we introduce large quantities of vision data in the SFT stage, along with agentic and reasoning text data. After warming up the vision projector, we use a two-stage SFT process because learning vision and text capabilities simultaneously is challenging. In the first stage, we train on a general mix of vision, reasoning, chat, long-context, safety, and agentic data. We then do a small second stage of SFT, where we change the blend composition to boost some capabilities the first stage did not learn adequately. Nemotron 3.5 Lightning is trained with a single SFT stage with the text-only SFT data.
**视觉数据。** 我们利用了来自 Nemotron 3 Nano Omni 训练语料的视觉数据,该语料涵盖通用视觉感知,以及跨图表、表格、OCR、多页文档、视觉定位、GUI 理解和视频理解的 VQA。它还涵盖数学、科学以及空间与时间关系上的视觉推理,多语言覆盖则集中于文档与 OCR 任务。我们使用 Qwen3.5 等开源模型新构建的数据扩充了该语料,以增强图表理解、有依据的感知、多跳推理、GUI/网页导航和多模态编程能力。数据生成流水线覆盖多样的视觉结构、领域、问题类型和推理任务。我们观察到,将推理轨迹组织为一系列明确的简洁步骤、且每一步都基于相关视觉证据,有助于教会模型系统化规划,并鼓励其针对视觉输入进行迭代验证。此外,我们应用自动化质量检查,以过滤歧义或缺乏视觉支持的问题、不完整的推理轨迹、幻觉标签或数值、错误的视觉或数值解读、过度重复以及无效的自我纠正。图 5 总结了该流水线。我们以开源 NeMo Data Designer 生成流水线配方⁵及配套开发说明⁶的形式,提供了该方法的一个图表理解实现。
**Vision Data.** We leveraged vision data from the Nemotron 3 Nano Omni training corpus, which covers general visual perception and VQA across charts, tables, OCR, multi-page documents, visual grounding, GUI understanding, and video understanding. It also covers visual reasoning over mathematics, science, and spatial and temporal relationships, with multilingual coverage concentrated in document and OCR tasks. We augmented this corpus with newly curated data using open-source models such as Qwen3.5 to strengthen chart understanding, grounded perception, multi-hop reasoning, GUI/web navigation, and multimodal coding. The data generation pipeline covers diverse visual structures, domains, question types, and reasoning tasks. We have observed that organizing the reasoning traces into an explicit sequence of concise steps, with each step grounded in relevant visual evidence, helps teach the model systematic planning and encourages iterative verification against the visual input. Additionally, we apply automated quality checks to filter ambiguous or visually unsupported questions, incomplete reasoning traces, hallucinated labels or values, incorrect visual or numerical interpretations, excessive repetition, and unproductive self-correction. Figure 5 summarizes the pipeline. We provide a chart-understanding implementation of this approach as an open-source NeMo Data Designer generation pipeline recipe⁵ and a companion development note⁶.
**智能体式视觉数据。** 视觉工具使用是智能体的一项基本能力。为教会模型获得这一智能体式能力,我们生成了一个覆盖不同领域的综合数据集。我们使用 Kimi K2.6 和 K3 生成智能体式数据集,重点针对高分辨率图像的裁剪、旋转、缩放、翻转等图像工具操作。对于视频理解,我们使用 Kimi K2.6 从现有视频问答对生成可执行的 SpatialClaw 演示,教会学生模型在作答前收集视觉证据、使用工具并解读观察结果。源池涵盖属性感知、时间理解、物体推理、OCR、计数和视频摘要。每个助手动作包含 Purpose、Reasoning、Next Goal 和可执行 Python Code。动作在持久化内核中运行,返回执行反馈和视觉观察。智能体可以检查帧、通过单独的视觉问答调用查询选定帧、执行 SAM3 分割,并在以 ReturnAnswer 终止前重建场景几何。我们保留正常完成、答案与参考匹配且所需工具成功执行的轨迹。结构化答案采用规范化匹配;其他自由形式答案则要求同一教师模型进行纯文本等价性判定,且报告置信度 \(\geq 0.90\)。后续过滤会剔除格式错误的动作、执行失败、超出显示预算以及缺少成功有依据动作的轨迹。我们将录制的对话和图像打包用于 SFT,去除冗余格式和消息。最终语料包含 14,169 条轨迹和 81,366 个助手动作。我们在表 7 中展示了使用 SpatialClaw 测试框架的性能。
**Agentic Vision Data.** Visual tool use is a fundamental ability for agents. To teach the model to gain this agentic ability, we generate a comprehensive dataset spanning different areas. We use Kimi K2.6 and K3 to generate agentic datasets with a focus on image tool operations such as crop, rotate, zoom, flip, etc. for high-resolution images. For video understanding, we use Kimi K2.6 to generate executable SpatialClaw demonstrations from existing video question–answer pairs, teaching the student to collect visual evidence, use tools, and interpret observations before answering. The source pools cover attribute perception, temporal understanding, object reasoning, OCR, counting, and video synopsis. Each assistant action contains Purpose, Reasoning, Next Goal, and executable Python Code. Actions run in a persistent kernel that returns execution feedback and visual observations. The agent can inspect frames, query selected frames through a separate visual question-answering call, perform SAM3 segmentation, and reconstruct scene geometry before terminating with ReturnAnswer. We retain normally completed trajectories whose answers match the references and whose required tools execute successfully. Structured answers use canonical matching; other free-form answers require same-teacher, text-only equivalence judging with reported confidence \(\geq 0.90\). Subsequent filtering rejects malformed actions, execution failures, exceeded display budgets, and trajectories lacking a successful grounded action. We package the recorded conversations and images for SFT, removing redundant formatting and messages. The final corpus contains 14,169 trajectories and 81,366 assistant actions. We show the performance with the SpatialClaw harness in Table 7.
基线不使用任何智能体式视觉训练。\(\Delta\) 表示 Ours 减去 Baseline。
Baseline uses no agentic vision training. \(\Delta\) denotes Ours minus Baseline.
对于推理监督微调,我们使用了与 Nemotron Ultra 相同的底层推理问题集。这些问题集涵盖数学、STEM、编程和数学证明。对于数学和 STEM 问题集,我们将先前的教师模型替换为 Kimi K3,并重新生成了思维链(CoT)和工具集成推理(TIR)响应轨迹。数学证明轨迹使用 DeepSeek V4 Pro 生成,而竞赛编程轨迹使用 GLM-5.2 生成。
For reasoning supervised fine-tuning, we used the same underlying reasoning problem sets as Nemotron Ultra. These problem sets span mathematics, STEM, coding, and mathematical proof. For the mathematics and STEM problem sets, we replaced the previous teacher with Kimi K3 and regenerated both chain-of-thought (CoT) and tool-integrated reasoning (TIR) response trajectories. Mathematical-proof trajectories were generated using DeepSeek V4 Pro, while competitive-programming trajectories were generated using GLM-5.2.
作为数据过滤的一部分,我们移除了短于 5,000 个 token 的推理回复,以使训练混合数据聚焦于包含充分展开的多步推理的样本。我们还观察到,由两个主要推理教师模型生成的数据在学习动态上存在差异。对于 Nemotron Ultra,DeepSeek V4 Pro 担任主要推理教师,在对所得训练轨迹完成单次遍历后,下游推理性能基本趋于平稳。在将 Nemotron 3.5 Super 的主要推理教师切换为 Kimi K3 后,性能在多次遍历重新生成的轨迹过程中持续提升。我们假设,Kimi K3 生成的推理更简洁、信息密度更高,因此较小的学生模型需要更多次接触才能将其内化。
As part of data filtering, we removed reasoning responses shorter than 5,000 tokens to focus the training mixture on examples containing sufficiently developed, multi-step reasoning. We also observed a difference in learning dynamics between data generated by the two primary reasoning teachers. For Nemotron Ultra, DeepSeek V4 Pro served as the primary reasoning teacher, and downstream reasoning performance largely plateaued after a single pass over the resulting training trajectories. After switching the primary reasoning teacher to Kimi K3 for Nemotron 3.5 Super, performance continued to improve over multiple passes through the regenerated trajectories. We hypothesize that the more concise and information-dense reasoning produced by Kimi K3 requires additional exposure for a smaller student model to internalize.
**对话与安全数据。** 我们大体沿用 Nemotron 3 Ultra 的相同配方;不过,我们用更新的 Qwen、Kimi 和 Gemma 模型回复刷新了对话数据集。对于安全数据,我们大体使用与 Nemotron 3 Ultra 相同的混合数据,并额外加入一些聚焦于双重风险 CBRNE 内容和人口统计偏差的数据。
**Chat and Safety Data.** We largely follow the same recipe as Nemotron 3 Ultra; however, we refresh our chat dataset with more recent Qwen, Kimi, and Gemma model responses. For safety data, we largely use the same blend as Nemotron 3 Ultra, plus some additional data focused on dual-risk CBRNE content and demographic biases.
**长上下文数据。** 我们使用了 Nemotron 3 Ultra 的 512K 长上下文数据。此外,我们合成了图结构数据,以教会模型通过推理追踪节点和边,而不是仅依赖复制和扫描。我们还发现,来自长时程搜索和编码任务的智能体轨迹可以提升长上下文泛化能力。最后,对于 Nemotron 3.5 Super,我们从 PG19 和 FinePDFs-Edu 两个数据集合成生成长上下文翻译数据,其中每个实例是一篇长文档,使用 DeepSeek V4 Flash 将源语言分块翻译为目标语言。有关详细的源语言和目标语言以及 token 数量,请参见表 29–30。为评估长翻译性能,我们使用 SEGALE 评估流水线。我们发现,同时加入强化位置追踪、检索、变异和输出的合成转导任务可以提升长翻译性能。长翻译结果见图 12。我们还加入了使用 NeMo Data Designer 生成的长上下文视觉数据,用于对跨度达数百页的文档进行多页和整文档理解。这些样本需要跨页定位并组合来自文本、表格、图表和示意图的证据。^7
**Long-Context Data.** We used the 512K long-context data from Nemotron 3 Ultra. In addition, we synthetically generated graph-structured data to teach the model to trace nodes and edges through reasoning, rather than relying solely on copying and scanning. We also found that agent trajectories from long-horizon search and coding tasks can improve long-context generalization. Finally, for Nemotron 3.5 Super, we synthetically generate long-context translation data from both the PG19 and FinePDFs-Edu datasets, where each instance is one long document translated in chunks from the source language to a target language using DeepSeek V4 Flash. For detailed source and target languages as well as token counts, please see Tables 29–30. To evaluate long-translation performance, we use the SEGALE evaluation pipeline. We found that long-translation performance was improved by also including synthetic transduction tasks that reinforce position tracking, retrieval, mutation, and output. Long translation results can be seen in Figure 12. We also include long-context vision data generated with NeMo Data Designer for multi-page and whole-document understanding over documents spanning up to hundreds of pages. These examples require locating and combining evidence from text, tables, charts, and diagrams across pages.^7
**训练配置。** 在 SFT 阶段,我们联合微调语言和视觉组件。训练使用最多 524,288 个 token 的打包序列,全局批大小为 32。学习率在 100 个打包样本等价物(约 9,100 条原始记录)上预热至 \(1\times 10^{-5}\),随后保持不变。我们使用 Adam 优化器,\(\beta_{1}=0.9\)、\(\beta_{2}=0.95\),权重衰减为 0.1,梯度裁剪阈值为 1.0,同时保留两个共享权重的 MTP 层,辅助损失缩放因子为 0.3。
**Training Config.** For the SFT stage, we jointly fine-tune the language and vision components. Training uses packed sequences of up to 524,288 tokens with a global batch size of 32. The learning rate warms up over 100 packed-sample-equivalents—approximately 9,100 original records—to \(1\times 10^{-5}\), then remains constant. We use Adam with \(\beta_{1}=0.9\), \(\beta_{2}=0.95\), weight decay 0.1, and gradient clipping at 1.0, while retaining two shared-weight MTP layers with an auxiliary-loss scaling factor of 0.3.
强化学习提升了 SFT 模型的通用能力,并为开发专门的教师模型提供了坚实基础。在此阶段,我们联合应用 RLVR 与 RLHF,将来自各环境的样本路由至相应的验证器或奖励模型进行打分。与 Nemotron 3 Ultra 相比,我们使用更新的数据集合刷新了 RL 训练数据,并在训练前进行奖励画像(reward profiling)。我们采用与 [其] 相同的训练算法,并辅以两项旨在缓解训练–推理不一致的优化:router replay 以及对 LM head 使用 FP32。这些优化共同将训练引擎与推理引擎之间的 KL 误差降低了一个数量级以上。
Reinforcement learning improves the general capabilities of the SFT model and provides a strong foundation for developing specialized teacher models. At this stage, we jointly apply RLVR and RLHF, routing samples from each environment to the corresponding verifier or reward model for scoring. Compared with Nemotron 3 Ultra, we refresh the RL training data with more recent collections and perform reward profiling before training. We use the same training algorithm as [it], augmented with two optimizations designed to mitigate train–inference mismatch: router replay and using FP32 for the LM head. Together, these optimizations reduce the KL error between the training and inference engines by more than an order of magnitude.
我们还引入了新的数据,例如 [某工作] 的竞赛编程 RL 环境,包含来自历届奥赛级别竞赛的 2,847 道训练题和 372 道验证题,每题均配有经过验证的可执行评测器。生成的 C++ 解答会被编译并针对题目专属测试执行,获得满分时给予二值奖励 1,否则为 0。
We also incorporate new data such as the competitive-programming RL environment of [reference], comprising 2,847 training problems and 372 validation problems from past Olympiad-level competitions, each paired with a validated executable evaluator. Generated C++ solutions are compiled and executed against problem-specific tests, receiving a binary reward of 1 for full credit and 0 otherwise.
为提升金融任务中的工具使用与跨文档推理能力,我们在 NeMo Gym 之上构建了一个金融智能体环境,针对 Vals.ai Finance Agent 基准所衡量的能力。该环境在轨迹生成、监督微调(SFT)、强化学习(RL)和评估过程中提供相同的工具——SEC 文件检索、文档解析、证据检索以及最终有来源支撑的提交——从而避免各阶段之间的格式与工具不匹配。我们采用两条合成数据生成(SDG)路线。文档锚定路线从标普 500 公司的 SEC 10-K、10-Q 和 8-K 文件中采样窗口化上下文,用教师模型生成分析师风格的问题,并通过多种子投票与 GenSelect 验证可回答性和答案质量。智能体路线针对对照 Vals.ai 类别分类法进行覆盖审计时发现的缺口,将源文档之间的关系——报告期、表单类型和公司——设为显式的采样变量。经过验证的问题由教师模型在同一环境中重新求解,以生成经过验证的工具使用轨迹,并根据测得的难度将其路由至 SFT 或 RL 数据池。
To improve tool use and cross-document reasoning on financial tasks, we built a finance-agent environment on top of NeMo Gym targeting the capabilities measured by the Vals.ai Finance Agent benchmark. The environment provides the same tools—SEC filing search, document parsing, evidence retrieval, and final source-supported submission—during trajectory generation, supervised fine-tuning (SFT), reinforcement learning (RL), and evaluation, avoiding format and tool mismatch across stages. We use two synthetic data generation (SDG) tracks. The document-grounded track samples windowed contexts from SEC 10-K, 10-Q, and 8-K filings of S&P 500 companies, generates analyst-style questions with a teacher model, and verifies answerability and answer quality using multi-seed voting and GenSelect. The agentic track targets gaps found in a coverage audit against the Vals.ai category taxonomy by making the relationship among source documents—reporting period, form type, and company—an explicit sampling variable. Validated questions are re-solved by teacher models in the same environment to produce verified tool-use trajectories, and are routed to the SFT or RL pool according to their measured difficulty.
为支持在广泛而多样的环境集合上进行训练,我们使用 8,192 的全局批大小,每个样本生成 16 条 rollout。最大生成长度最初设为 80K token,随后提升至 128K。
To support training across a broad and diverse set of environments, we use a global batch size of 8,192, with 16 rollouts generated per sample. The maximum generation length is initially set to 80K tokens and later increased to 128K.
我们从第 120 步的 text-RLVR 检查点初始化智能体式教师(Agentic Teacher),并在智能体任务上以完整的端到端 GRPO 继续训练。在智能体式教师的训练过程中,我们不使用任何单步环境,而是选择在沙箱、完全可执行的智能体式 Terminal、SWE 和 Search 任务上训练端到端的智能体式教师。
We initialize the Agentic Teacher from the step-120 text-RLVR checkpoint and continue training with full end-to-end GRPO on agentic tasks. We do not use any single-step environments during the training of the agentic teacher; rather, we opt to train the end-to-end agentic teacher on sandbox, fully executable agentic Terminal, SWE, and Search tasks.
我们使用 64 个提示,每个提示生成 16 次,总批大小为 1024。在智能体式教师的训练中,我们将最大轨迹年龄保持为 1。每次 rollout 中,每个任务的最大超时时间设为 2 小时。我们针对第 120 步的 text-RLVR 检查点对任务进行了难度分析。
We use 64 prompts and 16 generations per prompt, giving a total batch size of 1024. We keep a max trajectory age of one for the training of the agentic teacher. Each task was given a maximum timeout of 2 hours per rollout. Tasks were profiled for difficulty for the step-120 text-RLVR checkpoint.
我们从第 120 步的 text-RLVR 检查点初始化推理教师,并使用相同的异步 GRPO 框架继续训练。主要区别在于训练数据混合与生成预算。我们将宽泛的 RLVR 混合数据替换为聚焦推理的课程,共 25,600 个样本,每个包含 256 条提示的批次中,90% 为科学问题、5% 为数学问题、5% 为竞赛编程问题。整个训练过程中,最大生成长度设为 131,072。
We initialize the reasoning teacher from the step-120 text-RLVR checkpoint and continue training with the same asynchronous GRPO framework. The main differences are the training mixture and the generation budget. We replace the broad RLVR mixture with a reasoning-focused curriculum of 25,600 examples, with each 256-prompt batch containing 90% science, 5% mathematics, and 5% competitive-coding problems. The maximum generation length is set to 131,072 throughout training.
每次更新使用 256 条提示,每条提示生成 16 个回复,总批次大小为 4,096。与初始 RLVR 设置相比,我们将最大轨迹年龄从一次更新增加到两次更新。学习率保持为 \(4 \times 10^{-6}\)。数学与 STEM 奖励采用答案验证,并以 DeepSeek-V4-Flash^{10} 作为 LLM 评判器,而编程奖励采用基于执行的测试。我们选择经过 80 次更新后的检查点作为 MOPD 阶段的推理教师。
We use 256 prompts per update and 16 responses per prompt, giving a total batch size of 4,096. We increase the maximum trajectory age from one to two updates compared to the initial RLVR setup. The learning rate remains \(4 \times 10^{-6}\). Mathematics and STEM rewards use answer verification with DeepSeek-V4-Flash^{10} as the LLM judge, while coding rewards use execution-based tests. We select the checkpoint after 80 updates as the reasoning teacher for the MOPD stage.
我们在视频问答推理数据和 SA-V 目标跟踪数据的混合数据上训练一个视频教师模型,其中视频问答推理数据是从 Nemotron 3 Nano Omni 的视频训练子集中精心筛选得到的。视频问答示例包含针对 64 个有序帧的多选题,这些帧被作为单个视频处理。SA-V 示例包含框、点和指代目标跟踪任务,其帧被作为有序多图像输入处理。这种混合旨在同时提升互补的高层视频推理能力(例如时间、属性、动作和对象)和细粒度的时空跟踪能力。
We train a video teacher on a mixture of video-QA reasoning data, which is carefully curated from the video training subset of Nemotron 3 Nano Omni, and SA-V object-tracking data. Video-QA examples contain multiple-choice questions over 64 ordered frames, which are processed as a single video. SA-V examples contain box, point, and referring-object tracking tasks whose frames are processed as ordered multi-image inputs. This mixture targets complementary high-level video-reasoning (e.g., temporal, attribute, action, and object) and fine-grained spatiotemporal-tracking capabilities.
两个数据源都使用确定性的、可验证的任务奖励,而不是 LLM 评判器。视频问答使用基于多选题答案正确性的二元奖励。对于 SA-V,模型在请求的帧上生成结构化轨迹,验证器根据定位质量、对象可见性准确度以及预测轨迹的有效性和完整性分配连续奖励。缺失、无效或多余的轨迹条目会受到惩罚。
Both sources use deterministic, verifiable task rewards rather than an LLM judge. Video-QA uses a binary reward based on multiple-choice answer correctness. For SA-V, the model generates structured tracks over the requested frames, and the verifier assigns a continuous reward based on localization quality, object-visibility accuracy, and the validity and completeness of the predicted track. Missing, invalid, or extraneous track entries are penalized.
我们额外应用两个辅助塑形信号。首先,如果响应没有恰好一个正确匹配的 <think>...</think> 块,则被视为格式错误:其任务奖励设为零,训练优势被覆盖为 \(-5\)。这防止了推理结构格式错误但答案正确的响应被强化。其次,我们仅对正确响应应用软长度偏好。推理长度不超过 1,024 个 token 时保留完整任务奖励;在 1,024 到 4,096 个 token 之间,奖励乘数从 \(1.0\) 线性下降到 \(0.95\);超过 4,096 个 token 时保持在 \(0.95\)。该偏好区间不会截断生成,并且对简洁、正确的推理最多引入 5% 的偏好。
We additionally apply two auxiliary shaping signals. First, a response without exactly one properly matched <think>...</think> block is considered malformed: its task reward is set to zero and its training advantage is overridden to \(-5\). This prevents an otherwise correct answer with malformed reasoning structure from being reinforced. Second, we apply a soft length preference only to correct responses. Reasoning up to 1,024 tokens retains the full task reward; between 1,024 and 4,096 tokens, the reward multiplier decreases linearly from \(1.0\) to \(0.95\); and beyond 4,096 tokens it remains at \(0.95\). This profile band does not truncate generation and introduces at most a 5% preference for concise, correct reasoning.
我们从 step-120 文本-RLVR 检查点初始化,并在 NeMo-RL 中使用异步 GRPO 进行训练。每次更新使用 128 个提示,每个提示 16 个响应,形成全局批次大小为 2,048 条轨迹。在评估策略时,我们重放 rollout 生成期间记录的 MoE 专家路由决策,以减少训练与生成之间的对数概率差异。MoE 路由器和视觉模块保持冻结。训练在 128 块 NVIDIA GB200 GPU 上使用恒定学习率 \(3\times 10^{-6}\)。
We initialize from the step-120 text-RLVR checkpoint and train using asynchronous GRPO in NeMo-RL. Each update uses 128 prompts and 16 responses per prompt, giving a global batch of 2,048 trajectories. We replay the MoE expert-routing decisions recorded during rollout generation when evaluating the policy, reducing train–generation log-probability discrepancies. The MoE router and vision modules remain frozen. Training uses a constant learning rate of \(3\times 10^{-6}\) on 128 NVIDIA GB200 GPUs.
我们选择 step 60 作为最终教师模型。在 256 帧推理协议下,它在 Video-MME 上达到 74.90,在相应的 256 帧推理协议下,在 Video-MME-v2 上达到 19.10,在 GOT-10k 上达到 52.59,在 Molmo2-VideoTrack 上达到 60.00。该检查点随后在 MOPD 期间用作视频专用教师模型。
We select step 60 as the final teacher. It achieves 74.90 on Video-MME under the 256-frame reasoning protocol, 19.10 on Video-MME-v2 under the corresponding 256-frame reasoning protocol, 52.59 on GOT-10k, and 60.00 on Molmo2-VideoTrack. This checkpoint is subsequently used as the video-specialized teacher during MOPD.
为促进复杂视觉任务中有效的工具使用,我们在精心筛选的内部数据混合集上训练了一个视觉工具调用教师模型。目标工具包括图像裁剪、缩放、旋转、翻转、差异检测、图像并排比较、叠加、目标计数和颜色分析,所有工具调用均以结构化 XML 表示。训练集由合成生成的多项选择题和数值题组成,这些题目需要借助一个或多个目标工具才能求解。我们进一步按难度过滤数据,保留那些在不使用可用工具时几乎无法正确求解的样本。我们以结果正确性作为主要奖励,并辅以工具使用奖励,以惩罚重复和格式错误的工具调用。我们从 text-RLVR 检查点初始化,并在 NeMo-RL 中使用异步 GRPO 进行训练。教师检查点的选择以 V*、HRBench8K 和 BabyVision 上的目标视觉工具使用基准准确率为指导。
To promote effective tool use in complex vision tasks, we train a vision tool-calling teacher on a curated mixture of internal data. The target tools include image cropping, zooming, rotation, flipping, difference detection, side-by-side comparison of images, overlays, object counting, and color analysis, with all tool calls represented as structured XML. The training set consists of synthetically generated multiple-choice and numerical questions that require one or more of the target tools to solve. We further filter the data by difficulty, retaining samples that are nearly impossible to solve correctly without using the available tools. We use outcome correctness as the primary reward, as well as an auxiliary tool-use reward to penalize duplicate and malformed tool calls. We initialize from the text-RLVR checkpoint and train using asynchronous GRPO in NeMo-RL. The teacher checkpoint selection is guided by the target vision tool use benchmark accuracy on V*, HRBench8K, and BabyVision.
为提升对视觉输入的推理能力,我们在一个经过精心筛选的混合任务集上训练,涵盖图表理解、文档与表格推理、OCR、几何以及视觉推理任务。第一阶段使用 16,090 个经过画像的样本。第二阶段使用由 13,848 个样本组成的混合数据,其中包括来自 Chart 数据集的 8,366 个图表推理问题,以及 5,482 个侧重文档、表格和 OCR 理解的样本。我们审查了参考答案格式及其与奖励函数的兼容性,并从第二阶段混合数据中移除了完全重复的问题–答案对。我们以结果正确性作为训练奖励,并使用与任务相适应的数学、多项选择和字符串匹配验证器进行评估。
To improve reasoning over visual inputs, we train on a curated mixture of chart understanding, document and table reasoning, OCR, geometry, and visual reasoning tasks. The first stage draws from 16,090 profiled examples. The second stage uses a mixture of 13,848 examples, comprising 8,366 chart-reasoning questions from Chart datasets, together with 5,482 examples emphasizing document, table, and OCR understanding. We audit reference-answer formats and compatibility with the reward functions, and remove exact question–answer duplicates from the second-stage mixture. We use outcome correctness as the training reward, evaluated using task-appropriate mathematical, multiple-choice, and string-matching verifiers.
我们从已有的多模态 RLVR 检查点初始化,并在 NeMo-RL 中使用异步 GRPO 进行训练,每次更新使用 256 个提示,每个提示采样 16 个回复。第二阶段从第一阶段第 114 步的检查点出发,并使用全新的优化器进行初始化。我们采用在第二阶段混合数据上额外进行 20 次更新后得到的检查点。两个阶段均使用分布在 32 个计算节点上的 128 块 GPU。
We initialize from an existing multimodal RLVR checkpoint and train using asynchronous GRPO in NeMo-RL, with 256 prompts per update and 16 sampled responses per prompt. The second stage initializes from the first-stage checkpoint at step 114 with a fresh optimizer. We use the checkpoint obtained after 20 additional updates on the second-stage mixture. Both stages use 128 GPUs distributed across 32 compute nodes.
我们训练了一个对齐教师(alignment teacher),它具备更优的对话能力,并且与人类偏好良好校准。我们主要使用基于人类反馈的强化学习(RLHF)来训练该教师。首先,我们训练了 Nemotron 3 Ultra GenRM 的一个新变体,该变体在训练中加入了推理长度惩罚,以提升 token 效率。结合 MTP 推理,它能够显著加快训练速度。随后,我们以与 Nemotron 3 Ultra 类似的方式进行策略训练。然而,我们不再以循环形式进行自我比较,而是将所有策略回复与一个强开源模型 Gemma 4 31B-it 的固定回复进行比较。我们还使用各评分项(rubric)得分的均值来评估回复,而非使用总体得分。我们的训练数据混合方式与 Nemotron 3 Ultra 类似,但加入了新收集的偏好数据、更新后的模型身份与能力数据,并更加侧重于解决语言混用问题。我们将该 RLHF 训练与 Nemotron 3 Ultra 中的弃答(abstention)训练策略相结合,并使用少量推理长度惩罚。我们在内部对话基准上评估下游策略的改进。
We train an alignment teacher that has superior chat capabilities and is well calibrated with human preferences. We train this teacher primarily using RLHF. First, we train a new variant of the Nemotron 3 Ultra GenRM that is trained with a reasoning length penalty to be more token efficient. Combined with MTP inference, it enables substantially faster training. We then conduct policy training in a similar manner to Nemotron 3 Ultra. However, rather than doing self-comparison in a cyclical format, we instead compare all policy responses to a fixed response from a strong open-source model, Gemma 4 31B-it. We also use the mean of per-rubric scores to evaluate responses instead of an overall score. Our training blend is similar to Nemotron 3 Ultra but includes newly collected preference data, updated model identity and capabilities data, and has more emphasis on addressing language mixing. We mix this RLHF training with the abstention training strategy from Nemotron 3 Ultra and use a small amount of reasoning length penalty. We evaluate downstream policy improvements on our internal chat benchmark.
Nemotron 3.5 Super 在强化学习(RL)之后包含一个多教师同策略蒸馏(MOPD)阶段,该阶段将前几小节中专门教师的能力蒸馏到一个单一的学生模型中,类似于 Nemotron 3 Ultra 的开发;最后的 MTP 提升阶段(第 3.7 节)则在 MOPD 的输出上运行。然而,Nemotron 3.5 Lightning 没有 MOPD 阶段。在本节中,我们详细介绍了 Nemotron 3.5 Super 的 MOPD 过程。学生模型是第 3.2 节中经过 120 步后的 RL 检查点,也是教师模型分支出来的同一检查点。给定来自某个领域的提示,学生生成一个响应;对于每个标记 \(x\) 及其上下文 \(c\)(提示和前面的标记),咨询拥有该领域的教师以获得其概率 \(p_{T}(x\mid c)\),并更新学生以提高教师评价高于学生的标记的概率,降低教师评价低于学生的标记的概率。具体来说,设 \(\pi_{S}\) 为参数为 \(\theta\) 的学生,\(\pi_{T}\) 为领域教师,\(c_{t}\) 为学生响应 \(y\) 中标记 \(y_{t}\) 的上下文。学生在其自身响应的每个位置最小化反向 KL 散度 \(D_{\mathrm{KL}}(\pi_{S}(\cdot|c_{t})\,\|\,\pi_{T}(\cdot|c_{t}))\),使用标记局部更新,并在每次更新时将采样的上下文视为固定,
Nemotron 3.5 Super included a multi-teacher on-policy distillation (MOPD) stage after RL which distills the capabilities of the specialized teachers of the preceding subsections into a single student model similar to the development of Nemotron 3 Ultra; the final MTP boosting phase (Section 3.7) then runs on the MOPD output. Nemotron 3.5 Lightning, however, did not have an MOPD stage. In this section we detail the MOPD process for Nemotron 3.5 Super. The student is the RL checkpoint of Section 3.2 after 120 steps, the same checkpoint the teachers were branched from. Given a prompt from some domain, the student generates a response; for each token \(x\) with context \(c\) (the prompt and preceding tokens), the teacher that owns the domain is consulted for its probability \(p_{T}(x\mid c)\), and the student is updated to raise the probability of tokens the teacher rates higher than it does and to lower it for tokens the teacher rates lower. Concretely, let \(\pi_{S}\) be the student with parameters \(\theta\), \(\pi_{T}\) the domain teacher, and \(c_{t}\) the context of token \(y_{t}\) in the student’s response \(y\). The student minimizes the reverse KL \(D_{\mathrm{KL}}(\pi_{S}(\cdot|c_{t})\,\|\,\pi_{T}(\cdot|c_{t}))\) at every position of its own responses, using the token-local update and treating the sampled contexts as fixed during each update,
其中 \(\bar{\pi}_{S}\) 是更新前学生的冻结快照(参数 \(\bar{\theta}\)),\(\mathrm{sg}[\cdot]\) 表示停止梯度,期望是在从 \(\bar{\pi}_{S}\) 采样的新响应上,更新在 \(\theta=\bar{\theta}\) 处评估。每个生成的标记从单次教师前向传播中获得密集优势,无需验证器,也无需从教师采样;在学生和教师一致的地方直接贡献为零,因此信号集中在学生偏离专家教师的地方。在实践中,训练是异步的,响应可能来自前一个快照 \(\bar{\pi}_{S-1}\),即滞后一步;遵循 Ultra 配方,标记随后通过 \(\bar{\pi}_{S}/\bar{\pi}_{S-1}\) 进行重要性加权,权重在 \([0.2,5]\) 之外的标记被掩码。这些标记级权重校正了采样上下文中的动作概率;上下文分布不匹配和掩码会引入偏差。只有学生生成的标记进入损失。
where \(\bar{\pi}_{S}\) is the frozen snapshot of the student taken before the update (parameters \(\bar{\theta}\)), \(\mathrm{sg}[\cdot]\) denotes stop-gradient, and the expectation is over fresh responses sampled from \(\bar{\pi}_{S}\) with the update evaluated at \(\theta=\bar{\theta}\). Every generated token receives a dense advantage from a single teacher forward pass, with no verifier and no sampling from the teacher; the direct contribution is zero where student and teacher agree, so the signal concentrates where the student departs from the specialist. In practice training is asynchronous and responses may come from the previous snapshot \(\bar{\pi}_{S-1}\), i.e. one step stale; following the Ultra recipe, tokens are then importance-weighted by \(\bar{\pi}_{S}/\bar{\pi}_{S-1}\) and those with weights outside \([0.2,5]\) are masked. These token-level weights correct action probabilities at the sampled contexts; context-distribution mismatch and masking introduce bias. Only student-generated tokens enter the loss.
Nemotron 3.5 Super 的 MOPD 在 128 个 GB200 节点上分两个阶段运行。我们发现,在蒸馏下基准测试以不同速率提升,有些在几步内达到峰值然后随着进一步训练而下降。为了缓解这一点,阶段 1 偏向于提升快且后期容易退化的领域,阶段 2 偏向于提升较慢的领域。提示来自用于训练每个教师的环境,并路由到该教师;没有专家拥有的环境被路由到一个自锚点,即学生起始检查点的冻结副本,当学生仍与其匹配时不提供信号,并正则化偏离它的行为。
MOPD for Nemotron 3.5 Super is run in two stages on 128 GB200 nodes. We found that benchmarks improve at different rates under distillation, and that some peak within a few steps and then decline with further training. To mitigate this, stage 1 is weighted toward the domains that improve quickly and tend to regress later, and stage 2 toward the domains that improve more slowly. Prompts come from the environments used to train each teacher and are routed to that teacher; environments no specialist owns are routed to a self-anchor, a frozen copy of the student’s starting checkpoint, which provides no signal while the student still matches it and regularizes departures from it.
学生是 RL 检查点。它从智能体式、推理、对齐(RLHF)和视觉工具使用教师中蒸馏,提示份额分别为 35%、25%、20% 和 15%,加上 5% 路由到 RL 检查点作为自锚点。批次包含 512 个提示,每个提示一个 rollout,生成在 128K 上下文中上限为 48K 标记,学习率为 \(2\times 10^{-6}\)。我们取 30 步后的检查点作为阶段 1 模型。
The student is the RL checkpoint. It is distilled from the agentic, reasoning, alignment (RLHF) and vision tool-use teachers with prompt shares of 35%, 25%, 20% and 15%, plus 5% routed to the RL checkpoint as self-anchor. Batches contain 512 prompts with one rollout each, generation is capped at 48K tokens in a 128K context, and the learning rate is \(2\times 10^{-6}\). We take the checkpoint after 30 steps as the stage-1 model.
阶段 2 针对聊天质量(例如 LMArena 风格的评估)和视觉领域,这些在阶段 1 中比智能体式和推理基准提升更慢。阶段 1 模型是学生和自锚点。智能体式和推理教师被退役,其环境并入自锚点流,因此学生在这两个领域上被正则化向阶段 1 模型,而不是再次蒸馏。四个专家被蒸馏:对齐教师、视觉工具使用教师、视觉推理教师,以及第 3.4.1 节的视频推理和跟踪教师,份额分别为 30%、20%、20% 和 20%,加上 10% 自锚点。批次增加到 1,024 个提示,生成可以使用完整的 128K 上下文,学习率为 \(3\times 10^{-6}\),阶段运行 40 步,对其提示进行一次遍历。
Stage 2 targets chat quality (e.g. LMArena-style evaluations) and the vision domains, which improved more slowly in stage 1 than the agentic and reasoning benchmarks. The stage-1 model is the student and self-anchor. The agentic and reasoning teachers are retired and their environments folded into the self-anchor stream, so the student is regularized toward the stage-1 model on them rather than distilled again. Four specialists are distilled: the alignment teacher, the vision tool-use teacher, a vision-reasoning teacher, and the video reasoning and tracking teacher of Section 3.4.1, with shares of 30%, 20%, 20% and 20%, plus 10% self-anchor. Batches grow to 1,024 prompts, generation may use the full 128K context, the learning rate is \(3\times 10^{-6}\), and the stage runs for 40 steps, one pass over its prompts.
在第二阶段,蒸馏目标通过信任域策略蒸馏的近端教师构造进行平滑(另见):公式 2 中的教师被替换为冻结的混合分布 \(\alpha\,\pi_{T}+(1-\alpha)\,\bar{\pi}_{S}\),其中 \(\alpha=0.2\)。这将优势下界限制为 \(\log(1-\alpha)\approx-0.22\),因此教师认为极不可能的 token 会收到有界的负信号而非无界信号,一旦学生已经吸收了一轮教师的知识,这就能稳定训练。第一阶段使用 \(\alpha=1\)。
In stage 2 the distillation target is smoothed with the proximal-teacher construction of trust-region policy distillation (see also ): the teacher in Equation 2 is replaced by the frozen mixture \(\alpha\,\pi_{T}+(1-\alpha)\,\bar{\pi}_{S}\) with \(\alpha=0.2\). This bounds the advantage below by \(\log(1-\alpha)\approx-0.22\), so tokens the teacher considers very unlikely receive a bounded negative signal rather than an unbounded one, which stabilizes training once the student has already absorbed one round of teachers. Stage 1 uses \(\alpha=1\).
教师作为独立、非并置的推理集群运行(张量并行 8,专家并行 8),其规模根据每个领域的份额和序列长度进行调整,并在轨迹到达时进行评分,因此 rollout 生成、教师评分和学生优化在 NeMo-RL 中形成一个流水线。MTP 头在整个过程中与语言模型联合训练(第 3.7 节)。
Teachers run as independent, non-colocated inference fleets (tensor parallel 8, expert parallel 8) sized to each domain’s share and sequence length, and score trajectories as they arrive, so rollout generation, teacher scoring and student optimization form a pipeline in NeMo-RL. The MTP heads are trained jointly with the language model throughout (Section 3.7).
我们在两个时间点应用一套专门的、仅针对 MTP 的训练配方:首先从 Nemotron 3.5 Super SFT 检查点进行短暂的 warm start,然后在所有主干后训练阶段完成后再次应用。我们将后一阶段称为 MTP boosting。对于 Nemotron 3.5 Lightning,我们跳过 warm start,仅应用 MTP boosting。在两次运行中,主干保持冻结,仅更新 MTP 头参数。
We apply a dedicated MTP-only training recipe at two points: first as a short warm start from the Nemotron 3.5 Super SFT checkpoint, and again after all backbone post-training stages are complete. We refer to the latter phase as MTP boosting. For Nemotron 3.5 Lightning, we skip the warm start and apply only MTP boosting. In both runs, the backbone remains frozen and only the MTP head parameters are updated.
该配方包含两个互补组件。第一,我们继续应用第 2.1 节所述的隐藏状态混合策略。这进一步使 MTP 头暴露于类推理条件,减少训练–推理不匹配,并提高较长草稿长度下的接受率。第二,此前的 MTP 训练阶段使用交叉熵损失,而这些运行使用以冻结主干为教师的知识蒸馏损失。这种教师分布监督改善了 MTP 头与主干之间的对齐,在非贪婪采样下尤其有益。
The recipe has two complementary components. First, we continue applying the hidden-state-mixing strategy described in Section 2.1. This further exposes the MTP head to inference-like conditions, reducing the train–inference mismatch and improving acceptance at longer draft lengths. Second, whereas the preceding MTP training stages use cross-entropy loss, these runs use a knowledge-distillation loss with the frozen backbone as the teacher. This teacher-distribution supervision improves alignment between the MTP head and the backbone and is particularly beneficial under non-greedy sampling.
为了衡量 MTP 草稿质量,我们报告在 SPEED-Bench 定性划分上使用每样本接受长度(AL)的评估结果。
In order to measure MTP draft quality we report evaluation results on the SPEED-Bench qualitative split using per-sample acceptance length (AL).
我们使用相应的主干检查点生成 on-policy rollout。种子提示分别来自 Nemotron-Post-Training-Dataset-v2[11] 和 Nemotron-RL-Super-Training-Blends[12],以覆盖通用和智能体式输入。我们以 \(temp=1\) 采样 rollout,将序列上限设为 8K tokens,并仅在助手响应 token 上累积损失。
We generate on-policy rollouts using the corresponding backbone checkpoint. Seed prompts are drawn from Nemotron-Post-Training-Dataset-v2[11] and Nemotron-RL-Super-Training-Blends[12] to cover general-purpose and agentic inputs, respectively. We sample rollouts at \(temp=1\), cap sequences at 8K tokens, and accumulate the loss only over assistant-response tokens.
我们首先从 Nemotron 3.5 Super SFT 检查点训练 MTP 头 \(4k\) 步。与最终检查点运行相比,这一中间运行在明显更少的训练迭代后即进入平台期。当迁移到最终后训练检查点时,所得的 MTP 头保留了其接受长度改进的显著部分,如表 8 所示。因此,我们使用 SFT 训练的 MTP 头来初始化最终的 MTP boosting 阶段,并针对最终的 Nemotron 3.5 Super 主干再训练 \(25k\) 步。对于 Nemotron 3.5 Lightning,我们针对最终主干运行了 \(48k\) 步的 MTP boosting。Nemotron 3 Ultra 使用温度缩放的前向 KL 目标,而我们现在使用 LK 损失。
We first train the MTP head for \(4k\) steps from the Nemotron 3.5 Super SFT checkpoint. This intermediate run plateaus after substantially fewer training iterations than the final-checkpoint run. When transferred to the final post-trained checkpoint, the resulting MTP head retains significant part of its acceptance-length improvement, as shown in Table 8. We therefore use the SFT-trained MTP head to initialize the final MTP boosting phase and train it for an additional \(25k\) steps against the final Nemotron 3.5 Super backbone. For Nemotron 3.5 Lightning, we ran MTP boosting for \(48k\) steps against the final backbone. Whereas Nemotron 3 Ultra uses a temperature-scaled forward-KL objective, we now use the LK loss.
Boosted-MTP 头在每个草稿位置上均优于原始 MTP(图 6),且在更深的层位置上增益最大,因为这些位置的训练–推理不匹配最为严重:在推理时,这些位置以该头自身的隐藏状态为条件,而非以主干网络的隐藏状态为条件。这些逐位置增益转化为 SPEED-Bench 每个类别中更高的 AL(表 9),将 \(T=1\) 时的平均 AL 从 3.78 个被接受词元提升至 4.61 个被接受词元。
The Boosted-MTP head improves over the original MTP at every draft position (Figure 6), with the largest gains at deeper positions, where the train–inference mismatch is most severe: at inference, these positions condition on the head's own hidden states rather than on the backbone's. These per-position gains translate into higher AL in every SPEED-Bench category (Table 9), raising the average AL at \(T=1\) from 3.78 to 4.61 accepted tokens.
我们在涵盖智能体式能力、推理与知识、多模态、对话能力与指令遵循、长上下文理解以及多语言性能的综合基准套件上评估 Nemotron 3.5 Super。Nemotron 3.5 Super 与基线的所有评估结果均通过 Nemo Gym^13(https://github.com/NVIDIA-NeMo/Gym)收集。为保证可复现性,关于评估设置的更多细节可在 Nemo Gym 的 recipes 文件夹及相应的可复现性教程^14(https://github.com/NVIDIA-NeMo/Gym/blob/main/nemotron_recipes/super-3.5/reproducibility.md)中找到。
We evaluate Nemotron 3.5 Super on a comprehensive suite of benchmarks spanning agentic capabilities, reasoning and knowledge, multimodal, conversational ability and instruction following, long-context understanding, and multilingual performance. All evaluation results for Nemotron 3.5 Super and baselines were collected via Nemo Gym^13 (https://github.com/NVIDIA-NeMo/Gym). For reproducibility purposes, more details on the evaluation settings can be found in the Nemo Gym recipes folder and the corresponding reproducibility tutorial^14 (https://github.com/NVIDIA-NeMo/Gym/blob/main/nemotron_recipes/super-3.5/reproducibility.md).
以下基准尚未接入我们的开源工具,对于这些基准,我们使用了其官方开源实现,或使用我们计划未来开源的内部脚手架:BabyVision、CharXiv-RQ、MathVision、MMLongBench-Doc、OCR-Reasoning、OCRBench-v2、V*、Video-MME-v2。NV Chat Benchmark 是 NVIDIA 内部开发的对话基准。
The following benchmarks are not yet onboarded in our open-source tools, and for these we used either their official open-source implementation or otherwise an internal scaffolding that we plan to open source in the future: BabyVision, CharXiv-RQ, MathVision, MMLongBench-Doc, OCR-Reasoning, OCRBench-v2, V*, Video-MME-v2. NV Chat Benchmark is an internal chat benchmark developed at NVIDIA.
所有模型均在相同的评估设置下进行评估,包括智能体式资源(CPU 算力与超时、工具、搜索 API)、输入数据、提示模板、重复次数以及抽取/指标实现。温度、top_p 等推理参数以及最大 token 数直接取自相应模型卡的推荐值。
All models were evaluated under the same evaluation settings, including agentic resources (CPU compute and timeouts, tools, search APIs), input data, prompt templates, number of repeats, and extraction/metric implementation. Inference parameters such as temperature and top_p, and the maximum number of tokens, were directly extracted from the recommendations of the corresponding model cards.
我们使用 SWE-bench Verified、SWE-bench Multilingual 和 SWE-bench Pro 评估软件工程方面的智能体式能力;使用 Terminal-Bench 2.1 评估基于终端的任务执行;使用 \(\tau^{2}\)-Bench 的 Retail、Airline 和 Telecom 领域以及 \(\tau^{3}\)-Bench 的 Banking 领域评估对话式工具使用;使用 GDPVal 评估具有经济价值的真实职场任务;使用 BrowseComp 评估网页浏览与信息检索。所有 SWE-bench 变体均在 OpenCode 测试框架^15(https://github.com/anomalyco/opencode)中运行,处于禁用网页搜索与网页抓取的沙箱内,我们报告三次运行的平均解决率。Terminal-Bench 2.1 使用 Terminus 2 测试框架,取八次运行的平均值。所有 \(\tau\)-Bench 领域均取八次试验的平均值。\(\tau^{2}\)-Bench 各领域使用 Qwen3-235B-A22B-Instruct-2507 作为用户模拟器;在 Banking 领域中,智能体使用 BM25 和 grep 工具从银行知识库中检索,并由 GPT-5.4-mini 充当用户模拟器。
We evaluate agentic capabilities using SWE-bench Verified, SWE-bench Multilingual, and SWE-bench Pro for software engineering; Terminal-Bench 2.1 for terminal-based task execution; the Retail, Airline, and Telecom domains of \(\tau^{2}\)-Bench and the Banking domain of \(\tau^{3}\)-Bench for conversational tool use; GDPVal for economically valuable real-world workplace tasks; and BrowseComp for web browsing and information seeking. All SWE-bench variants run in the OpenCode harness^15 (https://github.com/anomalyco/opencode) inside a sandbox with web search and web fetch disabled, and we report the resolve rate averaged over three runs. Terminal-Bench 2.1 uses the Terminus 2 harness and is averaged over eight runs. All \(\tau\)-Bench domains are averaged over eight trials. The \(\tau^{2}\)-Bench domains use Qwen3-235B-A22B-Instruct-2507 as the user simulator; in the Banking domain the agent retrieves from the bank's knowledge base with BM25 and grep tools, and GPT-5.4-mini acts as the user simulator.
我们的 GDPVal 开源实现大致遵循 AA v2.1 方法,使用与 Python 及系统软件包方面相同的沙箱环境,以及 Stirrup 智能体。参考委员会包含 AA 排行榜中部分模型的子集,其参考文件系独立收集(DeepSeek V4 Flash 0731、DeepSeek V4 Pro、Inkling Small、GLM-5.1、Kimi K2.6、Nemotron 3 Ultra、Qwen3.6-35B-A3B、Qwen3.5-122B-A10B、Qwen3.5-397B-A17B、GPT-OSS-120b、Gemma 4 26B-A4B、Qwen3-30B-A3B-Thinking-2507)。评审委员会与 AA 方法中所述相同(GPT-5.6 Sol,中等推理;Gemini 3.8 Flash,高推理;以及 Claude Opus 5,高投入)。
Our open-source implementation of GDPVal loosely follows the AA v2.1 methodology, using the same sandboxing environment with respect to Python and system packages, and a Stirrup agent. The reference committee includes a subset of the models from the AA leaderboard for which the reference files were collected independently (DeepSeek V4 Flash 0731, DeepSeek V4 Pro, Inkling Small, GLM-5.1, Kimi K2.6, Nemotron 3 Ultra, Qwen3.6-35B-A3B, Qwen3.5-122B-A10B, Qwen3.5-397B-A17B, GPT-OSS-120b, Gemma 4 26B-A4B, Qwen3-30B-A3B-Thinking-2507). The judge committee is the same as the one described in the AA methodology (GPT-5.6 Sol, medium reasoning; Gemini 3.8 Flash, high reasoning; and Claude Opus 5, high effort).
BrowseComp 缺乏标准化的测试框架,其得分会因搜索提供商、浏览工具、上下文管理和步数预算的不同而显著变化。因此,我们在 NeMo Gym 中发布的一个单一开源测试框架上评估所有模型。该框架提供由 Exa(深度模式)支持的搜索和浏览工具,并将检索到的页面存储在每个样本的工作区中,可通过只读 shell 工具访问。上下文采用丢弃所有策略进行管理:当对话超过 token 阈值时,先前的轮次被丢弃,模型通过记忆工具在重置之间保持进度。我们还在基于 OpenCode 的测试框架(对工具配置做了细微更改)上报告 BrowseComp 结果,所有模型在相同的服务设置下运行,并观察到 Nemotron 3.5 Super 的性能也能迁移到第二个测试框架上。
BrowseComp lacks a standardized harness, and scores vary substantially with search provider, browsing tools, context management, and step budget. We therefore evaluate all models on a single open-source harness released in NeMo Gym. The harness provides search and browse tools backed by Exa (deep mode) and stores retrieved pages in a per-sample workspace accessible through a read-only shell tool. Context is managed with a discard-all policy: when the conversation exceeds a token threshold, prior turns are dropped and the model maintains progress across resets via a memory tool. We additionally report BrowseComp on an OpenCode-based harness (with minor tool-configuration changes) under identical serving settings for all models, and observe that Nemotron 3.5 Super’s performance transfers to this second harness as well.
我们使用 LiveCodeBench v6 评估竞赛编程中的代码生成,并使用 SciCode 评估科学研究编码。对于 LiveCodeBench,我们使用 2024 年 8 月至 2025 年 5 月间发布的 454 个问题,提示模型在不使用工具的情况下解决它们,并报告三次样本平均的 pass@1。SciCode 包含 65 个由科学家撰写的研究问题,并分解为 288 个子问题;我们在提示中包含科学家提供的背景信息,并对八次重复取平均。我们报告子任务准确率(生成的代码通过其测试的子问题比例)和问题准确率(所有子问题都通过的问题比例)。
We evaluate code generation using LiveCodeBench v6 for competitive programming and SciCode for scientific research coding. For LiveCodeBench, we use the 454 problems released between August 2024 and May 2025, prompt the model to solve them without tools, and report pass@1 averaged over three samples. SciCode contains 65 research problems written by scientists and decomposed into 288 subproblems; we include the scientist-provided background in the prompt and average over eight repeats. We report both subtask accuracy, the fraction of subproblems whose generated code passes its tests, and problem accuracy, the fraction of problems for which all subproblems pass.
我们使用 GPQA-Diamond 评估研究生级别的科学问题;使用 Humanity’s Last Exam(HLE)评估人类知识前沿的专家级问题;使用 MathArena Apex-Shortlist 评估竞赛数学;使用 CritPt 评估研究级物理推理;使用 AA-Omniscience 评估事实知识和抗幻觉能力。GPQA-Diamond 取八次重复的平均值,Apex-Shortlist 取 16 次重复的平均值。Apex-Shortlist 的答案通过符号等价性检查,且模型无法使用工具。对于 HLE,我们在此报告 2,158 个纯文本问题,包括无工具和有工具两种设置。在无工具设置下,答案由 GPT-5.6 Luna 使用官方 HLE 评判提示进行评分。在有工具设置下,Nemotron 3.5 Super 可以运行 Python 代码并搜索网络;此结果来自一个单独的内部测试框架。CritPt 使用官方 Artificial Analysis 评分 API 进行评分,对其 70 个问题中的每个问题使用五个样本。对于 AA-Omniscience,我们使用 600 个问题的公开集,取十次重复的平均值,并使用 GPT-5.6 Luna 评分。我们报告准确率和非幻觉率。
We evaluate mathematical and scientific reasoning using GPQA-Diamond for graduate-level science questions; Humanity’s Last Exam (HLE) for expert-level questions at the frontier of human knowledge; MathArena Apex-Shortlist for competition mathematics; CritPt for research-level physics reasoning; and AA-Omniscience for factual knowledge and hallucination resistance. GPQA-Diamond is averaged over eight repeats and Apex-Shortlist over 16. Apex-Shortlist answers are checked by symbolic equivalence, and the model has no tool access. For HLE we report the 2,158 text-only questions here, both without tools and with tools. Without tools, answers are graded by GPT-5.6 Luna using the official HLE judge prompt. In the with-tools setting, Nemotron 3.5 Super can run Python code and search the web; this result comes from a separate internal harness. CritPt is scored with the official Artificial Analysis grading API, using five samples for each of its 70 problems. For AA-Omniscience, we use the 600-question public set, average over ten repeats, and grade with GPT-5.6 Luna. We report accuracy and the non-hallucination rate.
我们评估多模态理解,涵盖视觉推理、图表、文档和 OCR、视频以及计算机使用。我们使用 BabyVision 进行基础视觉推理;使用 CharXiv 推理问题进行科学图表理解;使用 MathVision 进行视觉情境下的数学推理;使用 OCR-Reasoning 和 OCRBench v2(后者分别报告英文和中文分数)进行文本丰富图像理解;使用 V* 进行高分辨率图像中的细粒度视觉搜索;使用 Video-MME-v2 进行视频理解;使用 MMLongBench-Doc 进行长多页文档理解;使用完整的 HLE 集(包括基于图像的问题);以及使用 OSWorld-Verified 评估计算机使用智能体。除非另有说明,结果取三次重复的平均值。自由形式的答案由 LLM 评判器提取或评分:BabyVision、MathVision 和 OCR-Reasoning 使用 GPT-4o-mini,CharXiv 使用 Claude Opus 4.7,MMLongBench-Doc 使用 GPT-4o,V* 的答案提取使用 GPT-3.5-Turbo。OCRBench v2 和 Video-MME-v2 使用基于规则的评分。V* 在没有视觉搜索工具的情况下作为静态多项选择题进行评估。Video-MME-v2 在没有字幕的情况下进行评估。MMLongBench-Doc 每个文档最多提供 100 个渲染页面,每页一张图像。HLE(完整)涵盖所有 2,500 个问题,使用单一样本,由 GPT-5.6 Luna 评分。对于 OSWorld-Verified,我们在云托管的桌面虚拟机上运行 361 个不需要 Google Drive 访问的任务。智能体仅观察截图,通过 pyautogui 命令行动,每个任务最多 100 步。我们报告三次运行的平均任务分数,其中包括部分分数。
We evaluate multimodal understanding across visual reasoning, charts, documents and OCR, video, and computer use. We use BabyVision for basic visual reasoning; CharXiv reasoning questions for scientific chart understanding; MathVision for mathematical reasoning in visual contexts; OCR-Reasoning and OCRBench v2, the latter with separate English and Chinese scores, for text-rich image understanding; V* for fine-grained visual search in high-resolution images; Video-MME-v2 for video understanding; MMLongBench-Doc for long multi-page document understanding; the full HLE set, including image-based questions; and OSWorld-Verified for computer-use agents. Unless noted otherwise, results are averaged over three repeats. Free-form answers are extracted or graded by an LLM judge: GPT-4o-mini for BabyVision, MathVision, and OCR-Reasoning, Claude Opus 4.7 for CharXiv, GPT-4o for MMLongBench-Doc, and GPT-3.5-Turbo for answer extraction on V*. OCRBench v2 and Video-MME-v2 use rule-based scoring. V* is evaluated as static multiple choice without a visual search tool. Video-MME-v2 is evaluated without subtitles. MMLongBench-Doc provides up to 100 rendered pages per document, one image per page. HLE (full) covers all 2,500 questions with a single sample, graded by GPT-5.6 Luna. For OSWorld-Verified, we run the 361 tasks that do not require Google Drive access on cloud-hosted desktop virtual machines. The agent observes screenshots only, acts through pyautogui commands, and has up to 100 steps per task. We report the average task score, which includes partial credit, over three runs.
我们使用 AA-LCR 和 GraphWalks 评估长上下文理解。AA-LCR(v1.1)包含 100 个问题,需要对总计约 100K token 的真实文档集进行推理。我们取 16 次重复的平均值,并按照官方协议使用 GPT-5.6 Luna 对答案进行评分。GraphWalks 在提示中以边列表的形式向模型提供图,并要求给出广度优先搜索的结果或节点的父节点。我们包括所有上下文长度桶中的两种任务类型,并保留 1,150 个提示中适合 1M token 的 950 个;所有模型在相同的 950 个提示上评分。分数是预测节点集和参考节点集之间的 F1,来自单一样本。
We evaluate long-context understanding with AA-LCR and GraphWalks. AA-LCR (v1.1) contains 100 questions that require reasoning across sets of real-world documents totaling about 100K tokens. We average over 16 repeats and grade answers with GPT-5.6 Luna following the official protocol. GraphWalks gives the model a graph as an edge list in the prompt and asks for the result of a breadth-first search or for the parents of a node. We include both task types across all context-length buckets and keep the 950 of 1,150 prompts that fit within 1M tokens; all models are scored on the same 950 prompts. The score is the F1 between the predicted and reference node sets, from a single sample.
我们使用 IFBench 评估指令遵循能力,该基准衡量模型对 58 个新的、域外可验证输出约束的泛化程度。我们报告在严格验证下每个提示所满足约束的比例,并在提示和八个样本上取平均。NV Chat Benchmark 以成对比较中的胜率来衡量聊天偏好。它涵盖 3,713 个提示,每个提示都有一个来自人类偏好对战的对手回复。Gemini 3.1 Pro 以两种呈现顺序对每一对进行评判,并基于回复长度进行风格控制。此外,表 11 报告了源自人类成对偏好投票的 Text Arena Elo 评分,涵盖所有提示和仅英文提示,并分别给出有风格控制和无风格控制的结果。
We assess instruction following with IFBench, which measures how well models generalize to 58 new, out-of-domain verifiable output constraints. We report the fraction of constraints satisfied per prompt under strict verification, averaged over prompts and eight samples. NV Chat Benchmark measures chat preference as a win rate in pairwise comparisons. It covers 3,713 prompts, and each prompt has an opponent response drawn from human-preference battles. Gemini 3.1 Pro judges each pair in both presentation orders, with style control based on response length. In addition, Table 11 reports Text Arena Elo ratings derived from human pairwise preference votes, for all prompts and for English prompts only, with and without style control.
我们使用 MMLU-ProX 评估多语言知识与推理能力,并使用 WMT24++ 评估机器翻译能力。对于 MMLU-ProX,我们报告在十种语言上取平均的准确率:英语、德语、西班牙语、法语、意大利语、日语、中文、印地语、葡萄牙语和韩语。对于 WMT24++,我们将英语翻译为该集合中的九种非英语语言,并报告基于参考的 XCOMET-XXL 分数,在语言对上取宏平均。
We evaluate multilingual capabilities using MMLU-ProX for multilingual knowledge and reasoning and WMT24++ for machine translation. For MMLU-ProX, we report accuracy averaged over ten languages: English, German, Spanish, French, Italian, Japanese, Chinese, Hindi, Portuguese, and Korean. For WMT24++, we translate from English into the nine non-English languages of this set and report reference-based XCOMET-XXL scores, macro-averaged over language pairs.
Nemotron 3.5 Super 针对三种推理模式进行训练:关闭推理、常规推理与中等投入推理。常规推理模式与中等投入推理模式可选择与推理时预算控制结合使用。这些控制方式的组合提供了灵活性,覆盖了精度—效率权衡的完整谱系,以满足客户在各种应用场景中的需求。
Nemotron 3.5 Super is trained for three reasoning modes: reasoning-off, regular and medium-effort. The regular and medium-effort reasoning modes have the option to be used in conjunction with inference-time budget control. These combinations of controls provide flexibilities that cover the entire spectrum of accuracy-efficiency trade-off to meet customers' needs in various application scenarios.
中等投入推理模式在 SFT 阶段引入,其数据由 Kimi K3 的低投入模式生成,随后在 RL 阶段进行优化。沿用与 Nemotron 3 Ultra 相同的配方,对一小部分 SFT 数据的推理轨迹按随机推理预算进行截断,同时保持响应不变,并在训练损失中屏蔽 </think> token,以使模型为推理时预算控制做好准备。约 4% 的 RL 训练提示处于中等投入模式,覆盖数学、STEM 与编程,并对其 RL 奖励施加基于长度的调整。这与训练 Nemotron 3 Ultra 和 Super 所用的配方相同,且投入级别优化的效果可泛化到数学、STEM 与编程之外的多种任务。在 Artificial Analysis Intelligence Index 的各基准上平均而言,Nemotron 3.5 Super 的中等投入模式所用 token 比常规模式少约 2.2 倍,代价是精度下降约 9%。
The medium-effort reasoning mode is introduced during the SFT stage, with data generated by Kimi K3's low-effort mode, and later optimized during the RL stage. Following the same recipe as Nemotron 3 Ultra, the reasoning traces of a small fraction of SFT data are truncated to random reasoning budgets while the responses remain the same and with </think> tokens masked from training loss, to prepare the model for inference-time budget control. Approximately 4% of the RL training prompts are in the medium-effort mode and they cover math, STEM and coding, and length-based adjustments are applied on the RL rewards for them. This is the same recipe that has trained Nemotron 3 Ultra and Super and the effects of effort-level optimization generalize to a variety of tasks beyond math, STEM and coding. Nemotron 3.5 Super's medium-effort mode uses around 2.2X less tokens than the regular mode, averaged over the benchmarks in Artificial Analysis Intelligence Index, at the cost of approximately 9% drop in accuracy.
模型不应“过拟合”到单一智能体框架上,这一点很重要。我们在 SWE-bench Pro 和 Terminal-Bench 2.1 上,跨六个智能体框架对 Nemotron 3.5 Super 进行了评估。图 8 比较了各框架标准配置下的观测得分。Claude Code 在 SWE-bench Pro 上取得最高观测得分(49.45%),而 Codex 在 Terminal-Bench 2.1 上取得最高观测得分(61.80%)。各框架的排序在两个基准之间并不一致,这凸显了在报告模型结果的同时报告所用框架及其配置的重要性。
It is important for the model to not "overfit" into one single harness. We evaluate Nemotron 3.5 Super across six agent harnesses on SWE-bench Pro and Terminal-Bench 2.1. Figure 8 compares the observed scores under each harness's standard configuration. Claude Code achieves the highest observed score on SWE-bench Pro (49.45%), while Codex achieves the highest observed score on Terminal-Bench 2.1 (61.80%). The ordering of harnesses differs between benchmarks, highlighting the importance of reporting the harness and its configuration alongside model results.
我们通过并行智能体及随后的自我验证,进一步提升了最终模型的智能体式性能。对于每个任务,同一后训练智能体的 \(N\) 个独立实例尝试该任务,产生一个候选轨迹池。这些智能体可以并行执行,无需智能体间通信。然后,同一个检查点作为验证器,使用 LLM-as-a-Verifier(LAV)选择一个候选。这既不需要额外的参数更新,也不需要单独训练的验证器。
We further improve the final model's agentic performance through parallel agents followed by self-verification. For each task, \(N\) independent instances of the same post-trained agent attempt the task, producing a pool of candidate trajectories. These agents can execute in parallel without inter-agent communication. The same checkpoint then serves as a verifier, selecting one candidate using LLM-as-a-Verifier (LAV). This requires neither additional parameter updates nor a separately trained verifier.
为了说明选择过程,以 Terminal-Bench 2.1 为例,每个任务有五个候选轨迹。验证器接收任务描述和一对轨迹,并根据三个标准评估它们:规范遵循性、输出匹配和错误信号检测。这些标准评估智能体是否满足任务的明确要求、其观察到的终端输出是否支持成功完成,以及是否仍有未解决的错误。每次评估聚焦于一个标准,并为每个轨迹分配单独的等级分布。遵循 LAV,我们使用由单个 token 标签 A–T 表示的 20 级评分量表,从最好到最差排序。我们将每个轨迹的分数计算为这些等级的概率加权平均值,并归一化到 \([0,1]\),而不是选择单个离散等级。
To illustrate the selection process, consider Terminal-Bench 2.1, where each task has five candidate trajectories. The verifier receives the task description and a pair of trajectories, and evaluates them against three criteria: specification adherence, output match, and error signal detection. These criteria assess whether the agent meets the task's explicit requirements, whether its observed terminal output supports successful completion, and whether unresolved errors remain. Each evaluation focuses on one criterion and assigns a separate grade distribution to each trajectory. Following LAV, we use a 20-level grading scale represented by single-token labels A–T, ordered from best to worst. We compute each trajectory's score as the probability-weighted average over these grades, normalized to \([0,1]\), rather than selecting a single discrete grade.
对于每个比较对和标准,我们评估两种呈现顺序(一次轨迹 \(i\) 在前、\(j\) 在后,另一次位置互换;\(K=2\))。我们将得到的分数与原始候选身份对齐,并对两种顺序和所有三个标准取平均,以减轻呈现顺序偏差和评估噪声。因此,每个比较对获得对相同两个轨迹的六次评估,而无需生成额外的智能体 rollout。得到的对特定分数 \(R_{i}\) 和 \(R_{j}\) 使用 Bradley–Terry 模型 \(p_{ij}=1/(1+\exp(-(R_{i}-R_{j})))\) 转换为偏好概率。
For each compared pair and criterion, we evaluate both presentation orders (once with trajectory \(i\) first and \(j\) second, and once with their positions reversed; \(K=2\)). We align the resulting scores with the original candidate identities and average across both orders and all three criteria to mitigate presentation-order bias and evaluation noise. Each compared pair therefore receives six evaluations of the same two trajectories, without generating additional agent rollouts. The resulting pair-specific scores \(R_{i}\) and \(R_{j}\) are converted into a preference probability using the Bradley–Terry model \(p_{ij}=1/(1+\exp(-(R_{i}-R_{j})))\).
这些成对评估使用 LAV 的概率枢轴锦标赛(PPT)组织。对于五个 Terminal-Bench 候选,锦标赛首先在随机环中比较相邻候选,例如 \((1,2)\)、\((2,3)\)、\((3,4)\)、\((4,5)\) 和 \((5,1)\)。在其环比较中平均偏好最高的两个候选成为枢轴。然后,其余三个候选中的每一个与两个枢轴比较,并且两个枢轴相互比较;对于相同的定向比较,重用缓存的评估。最后,聚合两个阶段的偏好,并选择在其比较中平均偏好最高的候选。通过比较次数归一化考虑了枢轴参与更多比较的情况,并且五个候选中的任何一个最终都可能获胜。
These pairwise evaluations are organized using LAV's Probabilistic Pivot Tournament (PPT). For the five Terminal-Bench candidates, the tournament first compares neighboring candidates in a random ring, for example \((1,2)\), \((2,3)\), \((3,4)\), \((4,5)\), and \((5,1)\). The two candidates with the highest mean preference across their ring comparisons become pivots. Each of the remaining three candidates is then compared against both pivots, and the two pivots are compared with each other; cached evaluations are reused for identical directed comparisons. Finally, the preferences from both stages are aggregated, and the candidate with the highest mean preference across its comparisons is selected. Normalizing by the number of comparisons accounts for the pivots participating in more comparisons, and any of the five candidates can ultimately win.
我们在所有六个基准上应用相同的过程,对于 Terminal-Bench 2.1 和 DeepSWE 使用五个候选轨迹,对于 SWE-bench Verified、Pro、Multilingual 和 BrowseComp 使用三个。所有基准使用三个标准和 \(K=2\)。Terminal-Bench 和三个 SWE-bench 变体使用两个枢轴,而 DeepSWE 和 BrowseComp 使用一个枢轴。对于 BrowseComp,鉴于评估完整基准的成本,所有报告的指标使用用于我们内部评估跟踪的 400 问题子集。
We apply the same procedure across all six benchmarks, using five candidate trajectories for Terminal-Bench 2.1 and DeepSWE, and three for SWE-bench Verified, Pro, Multilingual, and BrowseComp. All benchmarks use three criteria and \(K=2\). Terminal-Bench and the three SWE-bench variants use two pivots, while DeepSWE and BrowseComp use one pivot. For BrowseComp, all reported metrics use the 400-problem subset used for our internal evaluation tracking, given the cost of evaluating the full benchmark.
如图 9 所示,选择机制在全部六个基准上均提升了观测准确率。Terminal-Bench 的准确率从 62.7% 提升至 71.9%,DeepSWE 从 24.6% 提升至 34.5%,BrowseComp 从 71.5% 提升至 79.8%——绝对增益分别为 9.2、9.9 和 8.3 个百分点。SWE-bench Verified、Pro 和 Multilingual 分别提升 1.1、2.4 和 1.9 个百分点。在 BrowseComp 子集上,选择机制弥补了 pass@1 与 oracle best-of-three 性能之间约 70% 的差距。这些结果表明,存在一条在推理阶段提升任务准确率的路径,它无需改变模型参数即可对后训练形成补充。
As shown in Fig. 9, selection improves observed accuracy across all six benchmarks. Terminal-Bench accuracy increases from 62.7% to 71.9%, DeepSWE from 24.6% to 34.5%, and BrowseComp from 71.5% to 79.8%—absolute gains of 9.2, 9.9, and 8.3 percentage points, respectively. SWE-bench Verified, Pro, and Multilingual improve by 1.1, 2.4, and 1.9 percentage points. On the BrowseComp subset, selection recovers approximately 70% of the gap between pass@1 and oracle best-of-three performance. These results demonstrate an inference-time path to higher task accuracy that complements post-training without changing model parameters.
来自许多合成和整理来源的后训练数据可能会吸收基准内容,不仅包括逐字复制,还包括字符串匹配无法捕捉的改写、翻译、增强和其他衍生样本。这种污染会夸大基准分数,并且由于相同的基准指导训练监控和检查点选择,它也可能导致最终模型更差。因此,我们对后训练数据进行了针对 30 个基准的检查(完整列表见附录 C.2),并删除了与基准问题相关的样本。这种数据侧检测补充了使用 CoDeC 的模型侧检测,后者探测训练模型以寻找基准的痕迹。该流程将作为 NeMo Curator 的一部分发布。
Post-training data drawn from many synthetic and curated sources can absorb benchmark content, not only as verbatim copies but also as paraphrases, translations, augmentations, and other derived samples that string matching misses. Such contamination inflates benchmark scores and, because the same benchmarks guide training monitoring and checkpoint selection, it can also result in a worse final model. We therefore performed checks of post-training data against 30 benchmarks (see the complete list in Appendix C.2) and removed samples related to benchmark questions. This data-side detection complements model-side detection with CoDeC, which probes a trained model for traces of a benchmark. The pipeline will be released as part of NeMo Curator.
该流程结合了两个简单的相似性信号(以高召回率选择候选匹配)和一个 LLM 评判器(精确验证每个候选)。每个训练样本经过四个阶段:
The pipeline combines two simple similarity signals, which select candidate matches with high recall, with an LLM judge, which verifies each candidate precisely. Every training sample passes through four stages:
归一化。样本被简化为其问题陈述,即基准问题可能泄漏的部分。在其数据集中重复出现的文本模板被剥离,以避免虚假匹配。
Normalization. The sample is reduced to its problem statement, the part into which a benchmark question could leak. Text templates that recur across its dataset are stripped to avoid spurious matches.
候选检索。使用两个信号将样本与每个基准问题进行比较。逐字重叠度量问题文本的\(n\)-gram 在样本中出现的数量。它检测逐字复制,即使在较长的文本中,但会遗漏任何改写。语义重叠度量两个文本部分之间的嵌入相似度。它有效捕捉改写,但可能因风格相似而产生偏差。两个信号都用于筛选最可能的污染候选。
Candidate retrieval. The sample is compared with every benchmark question using two signals. Verbatim overlap measures how many \(n\)-grams of the question’s text appear in the sample. It detects verbatim copies, even inside longer texts, but misses any rephrasing. Semantic overlap measures the embedding similarity between parts of the two texts. It effectively captures paraphrases, but can be biased by style similarity. Both signals are used to shortlist the most likely contamination candidates.
验证。每个入围的样本和基准问题对由 LLM 评判器验证。评判器将关系分类为污染(直接复制、改写、答案泄漏或方法泄漏)或良性(风格重叠或无污染),并决定其严重性。由于入围名单很小,即使强大的评判器也可以在此步骤中使用。
Verification. Each shortlisted pair of a sample and a benchmark question is verified by an LLM judge. The judge classifies the relationship either as contamination (direct copy, paraphrase, answer leakage, or method leakage) or as benign (style overlap or no contamination), and decides its severity. Because the shortlist is small, even a strong judge can be used in this step.
**移除。** 若判定器发现某样本相对于任一基准问题存在严重污染,则该样本会被移除。
**Removal.** A sample is removed if the judge finds it severely contaminated with respect to any benchmark question.
我们在合成污染数据上评估了该流水线的质量。为此,我们生成了 Terminal-Bench 2.1 与 GPQA Diamond 问题的 10,000 多个变体,这些变体由 13 种强度不同的增强方式产生。这些增强方式涵盖字符噪声、裁剪,以及由 LLM 生成的改写、翻译和抽象化。生成的变体与数量相近的同领域干扰样本混合。判定器分别正确识别出 87.5% 和 79.1% 的受污染样本,同时仅将单个干扰样本误标为受污染。关于评估设置的更多细节见附录 C.4。
We evaluated the quality of the pipeline on synthetic contamination. For this purpose, we generated over 10,000 variants of Terminal-Bench 2.1 and GPQA Diamond questions, produced by 13 augmentations of varying strength. Those augmentations ranged from character noise and cropping to LLM-generated paraphrases, translations, and abstractions. The generated variants were mixed with a similar number of same-domain distractors. The judge correctly identified 87.5% and 79.1% of the contaminated samples, respectively, while flagging only a single distractor as contaminated. More details on the evaluation setup can be found in Appendix C.4.
在后训练数据中,该流水线移除了 20,671 个样本(约 0.09%;其中 14,793 个 SFT 样本、5,878 个 RL 样本),涉及 30 个基准中 22 个基准的 1,808 个基准问题(6.0%)。在判定器处理的候选配对中,约 3% 被标记为严重污染。示例见附录 C.5。
From the post-training data, the pipeline removed 20,671 samples (around 0.09%; 14,793 SFT and 5,878 RL samples), matching 1,808 benchmark questions (6.0%) in 22 of the 30 benchmarks. Of the candidate pairs the judge worked on, around 3% were flagged as severe contamination. See Appendix C.5 for examples.
我们使用 ModelOpt17 对 Lightning 3.5 和 Super 3.5 VL 应用训练后量化(PTQ)和量化感知蒸馏(QAD),以相对于未量化的、训练后的 BF16 检查点,创造新的效率-精度前沿。
We apply post-training quantization (PTQ) and quantization-aware distillation (QAD) with ModelOpt17 on Lightning 3.5 and Super 3.5 VL to create a new efficiency-accuracy frontier relative to the unquantized, post-trained BF16 checkpoint.
配方开发始于定义边界:保守和激进。一些模型还可能定义非常激进的变体。这些边界用于在吞吐量-交互性帕累托曲线上,预测不同 SLA 下的运行时效率增益。除非另有说明,我们假设预缓存 ISL、新 ISL 和 OSL 分别为 51200、2048 和 1024。我们使用两个 SLA 来衡量量化候选与 BF16 之间的加速比:
Recipe development starts by defining bookends: conservative and aggressive. Some models might also define a very aggressive variant. These bookends are used to project runtime efficiency gains at various SLAs in the throughput-interactivity Pareto. Unless otherwise noted, we assume 51200, 2048, and 1024 as the pre-cached ISL, new ISL, and OSL, respectively. We measure speedups between a quantized candidate and BF16 using two SLAs:
在 50 tok/s/usr 的 SLA 下的吞吐量(tok/s/gpu)
Throughput (tok/s/gpu) at an SLA of 50 tok/s/usr
最大吞吐量(例如,通常为最高并发数)
Max throughput (e.g., usually highest concurrency)
部署用例也会影响配方选择。例如,Lightning 3.5 旨在在本地智能体式用例以及数据中心部署中表现出色。因此,我们对 Lightning 3.5 进行量化,以最小化权重获取并提高激活精度以保持准确性。无论 GPU SKU 如何,Lightning 3.5 都部署为 W4A16,而 Super 3.5 VL 的 NVFP4 检查点在 Blackwell 上部署为 W4A4,在非 Blackwell 平台上部署为 W4A16。
The deployment use case also influences recipe selection. For example, Lightning 3.5 is designed to excel in local, agentic use cases as well as datacenter deployments. We therefore quantize Lightning 3.5 to minimize weight fetch and raise activation precision to maintain accuracy. Lightning 3.5 deploys to W4A16 regardless of GPU SKU, while Super 3.5 VL’s NVFP4 checkpoint deploys W4A4 on Blackwell, and W4A16 on non-Blackwell platforms.
对 KV 缓存与 Mamba SSM 缓存进行量化,可减少 HBM 占用,并通过最小化数据传输提升解码吞吐量。KV 缓存随序列长度线性增长,而 Mamba SSM 状态的大小恒定。Super 3.5 VL 新增了对量化 NVFP4 KV 缓存的消融实验。我们在 §4.5 中表明,将 KV 缓存量化为 NVFP4 并与量化 SSM 结合,可满足验收标准。
Quantizing the KV and Mamba SSM caches reduces HBM footprint and improves decode throughput by minimizing data transfer. The KV cache grows linearly with sequence length, whereas the Mamba SSM state is constant in size. New to Super 3.5 VL are ablations on quantized NVFP4 KV cache. We show in §4.5 that quantization of the KV cache to NVFP4 in combination with a quantized SSM meets the acceptance criteria.
最后,我们的目标是在满足以下精度标准的同时,实现最高效率与最低 HBM 占用:
Finally, we aim for the highest efficiency and lowest HBM footprint while satisfying the following accuracy criteria:
中位精度恢复率达到 99% 或更高,其定义为多次重复实验中 NVFP4 与 BF16 评测分数之比的中位数。
Median accuracy recovery of 99% or higher, defined as the median of the ratios of NVFP4 to BF16 eval scores across several repeats.
相对于 BF16,冗长度增加不超过 5%。
Verbosity increase of no more than 5% relative to BF16.
量化后的 Lightning 3.5 与 Super 3.5 VL 的评测结果分别见 §4.9 和 §4.10。
Evaluation results for quantized Lightning 3.5 and Super 3.5 VL are shown in §4.9 and §4.10, respectively.
我们探索了三种预定义的精度配方(表 12)。发布的检查点采用激进配方,MoE 与共享专家 GEMM 使用 NVFP4,注意力机制投影与 Mamba 线性层使用 FP8。我们还探索了 AutoQuant,作为另一种由敏感度驱动的混合精度方案。
We explored three predefined precision recipes (Table 12). The released checkpoint uses the aggressive recipe, with NVFP4 MoE and shared-expert GEMMs and FP8 attention projections and Mamba linear layers. AutoQuant was explored as an alternative sensitivity-driven mixed-precision approach.
a 探索阶段曾考虑对共享专家使用 FP8;保守 BPE 估计假设所有共享专家均使用 NVFP4。b FP32 指路由器计算,而非检查点存储;BPE 在每一列中均使用以 BF16 存储的路由器权重。c 通过将每一列的权重策略应用于对应的主干与 MTP 模块进行估计;MTP 专用的融合投影与归一化参数保持 BF16。发布的 MTP 还保留 BF16 注意力机制权重;其存储量直接计入实测 BPE。发布的运行时使用 FP32 卷积缓存与 FP16+SR SSM 缓存。
a FP8 considered for the shared expert during exploration; the Conservative BPE estimate assumes NVFP4 for all shared experts. b FP32 denotes router computation, not checkpoint storage; BPE uses BF16-stored router weights in every column. c Estimated by applying each column's weight policy to corresponding backbone and MTP modules; MTP-specific fusion projection and normalization parameters remain BF16. The released MTP also retains BF16 attention weights; its storage is counted directly in the measured BPE. The released runtime uses an FP32 convolution cache and FP16+SR SSM cache.
我们将全模型 BPE 计算为 \(8S/N\),其中 \(S\) 为存储的张量负载(以字节计,包含量化缩放因子),\(N\) 为原始参数量(包含视觉、MTP 及未量化模块)。对于发布的检查点,\(S=74.987\) GB,\(N=124.453\) 十亿,得到 4.82 BPE;运行时缓存与激活值不计入其中。
We compute whole-model BPE as \(8S/N\), where \(S\) is stored tensor payload in bytes, including quantization scales, and \(N\) is the original parameter count, including vision, MTP, and unquantized modules. For the released checkpoint, \(S=74.987\) GB and \(N=124.453\) billion, giving 4.82 BPE; runtime caches and activations are excluded.
PTQ 确定候选配方的初始精度,QAD 用于进一步弥补量化差距。发布的检查点先采用预定义的激进 PTQ 配方,随后进行 200 次迭代的基于 SFT 的 QAD。相对于 BF16 权重参考(同样以 FP8 KV 缓存提供服务),其在 36 个基准上的精度恢复中位数为 99.62%。对于每个基准,我们分别对发布的检查点与 BF16 的已完成运行取平均,计算其得分比值,再在基准间取中位数。在 36 个带 token 测量的基准上,每个基准的平均冗长度(总量比值)增加了 0.44%。
PTQ establishes the initial accuracy of candidate recipes, and QAD is used to further recover the quantization gap. The released checkpoint uses the predefined aggressive PTQ recipe followed by SFT-based QAD at 200 iterations. Its median accuracy recovery is 99.62% across 36 benchmarks relative to the BF16-weight reference (also served with FP8 KV cache). For each benchmark, we first average completed runs separately for the released checkpoint and BF16, compute their score ratio, and then take the median across benchmarks. Across 36 benchmarks with token measurements, mean per-benchmark verbosity (Ratio of totals) increases by 0.44%.
发布的检查点使用 FP8 KV 缓存进行评估。我们的 KV 缓存消融实验显示,FP8 与 NVFP4 的精度相当。该检查点还包含量化并增强的 MTP。如前所述,该检查点采用预定义的激进配方并进行 QAD 恢复。
The released checkpoint was evaluated with an FP8 KV cache. Our KV cache ablations showed comparable accuracy with FP8 and NVFP4. It also includes the quantized, boosted MTP. As discussed, the checkpoint uses the predefined aggressive recipe with QAD recovery.
我们使用 NVIDIA Model Optimizer 应用后训练量化(PTQ),将 Nemotron 3.5 Super 量化为 NVFP4,以便在 NVIDIA GPU 上实现高效推理。该模型是一个 120B 参数、12B 激活的混合专家视觉语言模型,包含 C-RADIO 视觉编码器、每个 MoE 层具有 512 个路由专家的混合 Mamba/注意力机制语言主干,以及一个多词预测(MTP)头。
We apply post-training quantization (PTQ) using NVIDIA Model Optimizer to quantize Nemotron 3.5 Super to NVFP4 for efficient inference on NVIDIA GPUs. The model is a 120B-parameter, 12B-active mixture-of-experts vision-language model comprising a C-RADIO vision encoder, a hybrid Mamba/attention language backbone with 512 routed experts per MoE layer, and a multi-token prediction (MTP) head.
激活缩放因子从图像–文本对的校准集中学习,而非仅使用文本。图像经过视觉编码器,使语言模型能够观察到真实的输入分布,包括投影后的视觉 token。我们使用后训练阶段所用的 SFT 数据作为校准语料,并使用 1,024 个样本对模型进行校准。
Activation scales are learned from a calibration set of image–text pairs, rather than text alone. Images are passed through the vision encoder so that the language model observes the true input distribution, including the projected visual tokens. We use the SFT data employed during post-training as the calibration corpus and calibrate the model using 1,024 samples.
我们探索了多种并行策略以实现高效校准:流水线并行、数据并行(DP)× 专家并行(EP)以及 FSDP2。流水线并行一次仅有一个 GPU 处于活跃状态,因此我们选择 DP × EP。四个 rank 各处理 256 个不同样本,覆盖全部 1,024 个校准样本。512 个路由专家在各 rank 间均匀划分,MTP 专家也以相同方式划分。我们保持视觉编码器与投影器、稠密注意力机制层以及 Mamba 层为复制状态,因为专家构成了模型参数的大部分。
We explored several parallelism strategies for efficient calibration: pipeline parallelism, Data Parallel (DP) × Expert Parallel (EP), and FSDP2. Pipeline parallelism left only one GPU active at a time, so we chose DP × EP. Each of the four ranks processes 256 different samples, covering all 1,024 calibration samples. The 512 routed experts are split evenly across the ranks, and the MTP experts are split the same way. We keep the vision encoder and projector, dense attention layers, and Mamba layers replicated, since the experts make up most of the model's parameters.
我们在固定的 NVFP4 量化层级下比较了三种权重缩放因子选择方案:
We compared three weight-scale selection schemes at a fixed NVFP4 quantization tier:
Max:缩放因子由观测到的绝对最大值确定。
Max: The scale is determined from the observed absolute maximum.
MSE:通过遍历 FP8 缩放值来选择每个块的缩放因子,以最小化量化误差。
MSE: Each block scale is selected to minimize quantization error by sweeping over FP8 scale values.
MSE 4-over-6:基于 MSE 的缩放因子选择,但限制在更窄的搜索空间内。
MSE 4-over-6: MSE-based scale selection restricted to a narrower search space.
从评估结果(表 13)中我们观察到,与 Max 和 MSE 4-over-6 相比,MSE 算法在精度恢复方面更优。因此,我们在 PTQ 和 AutoQuant 方案中采用 MSE 校准。
From the evaluation results (Table 13) we observe that the MSE algorithm is superior in terms of accuracy recovery compared to the Max and MSE 4-over-6. Hence we use the MSE calibration for the PTQ and AutoQuant recipes.
我们使用 ModelOpt AutoQuantize(AutoQuant)以敏感度约束的混合精度搜索取代手工编写的层规则。对于每个可量化模块 \(i\) 和候选格式 \(q \in \{\mathrm{NVFP4}, \mathrm{FP8}, \mathrm{BF16}\}\),AutoQuant 根据由任务损失梯度的平方加权的量化诱导输出误差,估计敏感度 \(S_{i}(q)\)。随后求解
We use ModelOpt AutoQuantize (AutoQuant) to replace hand-authored layer rules with a sensitivity-constrained mixed-precision search. For each quantizable module \(i\) and candidate format \(q \in \{\mathrm{NVFP4}, \mathrm{FP8}, \mathrm{BF16}\}\), AutoQuant estimates a sensitivity \(S_{i}(q)\) from the quantization-induced output error weighted by the squared task-loss gradient. It then solves
其中 \(n_{i}\) 是权重数量,\(b(q_{i})\) 包含格式 \(q_{i}\) 的建模存储开销,\(\bar{b}\) 是目标有效位宽。必须共享同一部署格式的算子——例如 Q/K/V 投影以及 MoE 层内被路由的专家——构成单个搜索决策。视觉编码器、缓存量化器和固定的 MTP 配方被置于该搜索之外。
where \(n_{i}\) is the number of weights, \(b(q_{i})\) includes the modeled storage overhead of format \(q_{i}\), and \(\bar{b}\) is the target effective bitwidth. Operators that must share a deployment format—such as Q/K/V projections and the routed experts within an MoE layer—form a single search decision. The vision encoder, cache quantizers, and fixed MTP recipe are held outside this search.
敏感度分布高度不均匀:lm_head 占 NVFP4 敏感度的近一半,注意力机制和 Mamba 投影是次敏感的类别,而 MoE 专家相对稳健。候选分配遵循这一层级:将 NVFP4 集中在参数占主导的专家权重上,为敏感的注意力机制和 Mamba 投影保留 FP8 或 BF16,并为 lm_head 保留更高精度。
The sensitivity profile is highly nonuniform: the lm_head accounts for nearly half of the NVFP4 sensitivity, attention and Mamba projections are the next most sensitive families, and MoE experts are comparatively robust. The candidate assignments follow this hierarchy: they concentrate NVFP4 on the parameter-dominant expert weights, reserve FP8 or BF16 for sensitive attention and Mamba projections, and retain higher precision for the lm_head.
我们从训练后量化(PTQ)检查点初始化每个 QAD 学生模型。量化配置源自现有的量化参考检查点,该检查点来自 ModelOpt AutoQuant 或固定配方的 PTQ(第 4.2 和 4.3 节)。唯一有意的更改是 KV 缓存格式;量化参考检查点使用 FP8 KV 缓存,而我们的 PTQ 学生模型使用 NVFP4 KV 量化。因此,本小节中的所有结果均指使用 NVFP4 KV 缓存的部署。校准使用 512 个打包的多模态样本,变长记录打包至最多 524,288 个 token,打包缓冲区包含 5,000 个源样本。用于 QAD 学生模型初始化的 PTQ 工作流使用八个节点,每个节点配备四个 NVIDIA GB200 GPU(即总共 32 个 GPU)。完整工作流每个候选大约需要 50 分钟(即每个 PTQ 候选约 27 GB200 GPU 小时)。
We initialize each QAD student from a post-training quantized (PTQ) checkpoint. We derive the quantization configuration from an existing quantization-reference checkpoint, which comes from ModelOpt AutoQuant or fixed-recipe PTQ (Sections 4.2 and 4.3). The only intentional change is KV cache format; quantization-reference checkpoints use FP8 KV cache, whereas our PTQ student uses NVFP4 KV quantization. As such, all results in this subsection refer to deployment with NVFP4 KV cache. Calibration uses 512 packed multimodal samples, with variable-length records packed up to 524,288 tokens and the packing buffer contains 5,000 source samples. The PTQ workflow for QAD student initialization used eight nodes with four NVIDIA GB200 GPUs per node (i.e., 32 GPUs total). The complete workflow took approximately 50 minutes per candidate (i.e., about 27 GB200 GPU-hours per PTQ candidate).
我们研究了两种数据混合设置:后训练期间使用的 SFT 数据混合(100% SFT),以及 75% SFT 数据与 25% 教师生成的 RL 轨迹的混合。对于 RL 轨迹,我们使用 vLLM 为每个源记录采样 16 条轨迹。生成过程执行原始 RL 任务工作流,并保留环境和工具交互。我们屏蔽提示、对话历史、工具输出和预填充响应前缀 token。在我们的消融实验中,轨迹混合仅提供了适度的收益。
We study two data blend settings: the SFT data blend employed during post-training (100% SFT), and a mixture of 75% SFT data with 25% teacher-generated RL rollouts. For the RL rollouts we sample 16 trajectories per source record with vLLM. Generation executes the original RL task workflows and retains environment and tool interactions. We mask out prompt, conversation-history, tool-output, and prefilled response-prefix tokens. The rollout mixture provided only modest benefits in our ablations.
每个 QAD 实验都从其校准的 PTQ 学生检查点开始,并使用全新的优化器。我们仅优化启用量化器的语言模型 GEMM 的权重。所有其他参数保持冻结,包括 token 嵌入、未量化的语言模型参数、归一化参数、视觉编码器和投影器以及 MTP 模块。路由器偏置更新被显式禁用。量化配置和校准的量化器统计量从 PTQ 检查点恢复,而不是在 QAD 期间重新校准。我们使用 KL 散度作为目标。我们禁用主硬目标交叉熵损失、MTP 交叉熵损失和 MoE 负载均衡辅助损失,并且不使用中间特征蒸馏损失。优化使用分布式 AdamW,参数为 \(\beta_{1}=0.9\)、\(\beta_{2}=0.95\)、\(\epsilon=10^{-8}\)、权重衰减 \(0.1\),以及全局范数 \(1.0\) 的梯度裁剪。学习率在 9,100 个原始源记录上从零线性增加到 \(5\times 10^{-6}\),此后保持不变。我们在 128 个节点上运行 QAD,每个节点配备四个 NVIDIA GB200 GPU(总共 512 个 GPU)。对于每个量化候选,100 次迭代的 QAD 大约需要 2 小时(即 1024 GB200 GPU 小时)。
Each QAD experiment starts from its calibrated PTQ student checkpoint with a fresh optimizer. We optimize only the weights of language-model GEMMs with enabled quantizers. All other parameters remain frozen, including token embeddings, unquantized language-model parameters, normalization parameters, the vision encoder and projector, and the MTP module. Router-bias updates are explicitly disabled. The quantization configuration and calibrated quantizer statistics are restored from the PTQ checkpoint rather than recalibrated during QAD. We use KL divergence as our objective. We disable the main hard-target cross-entropy loss, MTP cross-entropy loss, and MoE load-balancing auxiliary loss and don’t use intermediate-feature distillation loss. Optimization uses distributed AdamW with \(\beta_{1}=0.9\), \(\beta_{2}=0.95\), \(\epsilon=10^{-8}\), weight decay \(0.1\), and gradient clipping at a global norm of \(1.0\). The learning rate increases linearly from zero to \(5\times 10^{-6}\) over 9,100 original source records and remains constant thereafter. We run QAD on 128 nodes with four NVIDIA GB200 GPUs per node (512 GPUs total). QAD for 100 iterations took about 2 hours (i.e., 1024 GB200 GPU-hours) for each of our quantization candidates.
我们评估了三种量化配方(我们按照上述描述为其生成了 PTQ 和 QAD 检查点):a) 4.8 BPE NVFP4 配方(基于相应 AutoQuant 运行选择的量化配置),b) 4.9 BPE NVFP4 配方(类似地,基于 4.8 BPE 目标 AutoQuant),c) 固定的 4.82 BPE NVFP4 配方。
We evaluated three quantization recipes (for which we generated PTQ and QAD checkpoints as described above): a) a 4.8 BPE NVFP4 recipe (based on the quantization configuration selected by the corresponding AutoQuant run), b) a 4.9 BPE NVFP4 recipe (similarly, based on a 4.8 BPE target AutoQuant), c) a fixed 4.82 BPE NVFP4 recipe.
下面,我们展示 4.8 BPE 配方的结果(准确性和冗长度);具体来说,包括 PTQ 检查点以及 200、400、600 次迭代的 QAD 检查点,涵盖 100% SFT 混合和 75/25% SFT/RL 混合两种情况。此初始评估扫描基于 BF16 对应版本的早期(即最终之前)版本。表 14 和表 15 将量化模型与该 BF16 基线进行比较。PTQ 保持了 BF16 准确率的中位数 97.68%,同时平均生成的完成 token 数减少了 0.22%。QAD 持续缩小了大部分剩余准确率差距,评估配置达到了 98.36% 至 99.18% 的中位数分数恢复。在使用 100% SFT 数据进行 200 次 QAD 迭代后,我们获得了最佳的整体准确性-冗长度权衡(中位数准确率恢复 99.18%——比 PTQ 高 1.50 个百分点——而总完成 token 数相对于 BF16 仅增加 1.11%)。额外的 QAD 迭代并非一致有益。对于 100% SFT 实验,100 次迭代的检查点提供了最强的整体准确性-冗长度权衡。75/25% SFT/RL 实验在中间迭代检查点表现最佳。总体而言,我们的结果倾向于使用 100% SFT 数据进行较短的 QAD 训练,并表明最佳训练时长取决于所选的数据混合。
Below, we show the results (accuracy and verbosity) for the 4.8 BPE recipe; specifically, the PTQ checkpoint and the QAD checkpoints for 200, 400, 600 iterations, for both the 100% SFT blend and the 75/25% SFT/RL blend cases. This initial evaluation sweep was based on an earlier (i.e., before final) version of the BF16 counterpart. Tables 14 and 15 compare the quantized models against that BF16 baseline. PTQ preserves a median 97.68% of the BF16 accuracy while producing 0.22% fewer completion tokens on average. QAD consistently closes much of the remaining accuracy gap, with the evaluated configurations achieving between 98.36% and 99.18% median score recovery. We obtain the best overall accuracy–verbosity tradeoff after 200 QAD iterations using 100% SFT data (99.18% median accuracy recovery - 1.50 percentage points above PTQ - with only a 1.11% increase in aggregate completion-token count relative to BF16). Additional QAD iterations were not uniformly beneficial. For the 100% SFT experiments, the 100 iterations checkpoint provides the strongest overall accuracy–verbosity tradeoff. The 75/25% SFT/RL experiments perform best at the intermediate iterations checkpoint. Overall, our results favor shorter QAD training with 100% SFT data and show that optimal training duration depends on the selected data mixture.
我们根据上述观察结果,针对最终后训练的 BF16 检查点调整了 QAD 实验。具体而言,我们将检查点间隔减少到 100 次迭代,仅使用 100% SFT 混合,并在 200、300 和 400 次迭代时评估 QAD 检查点。我们对之前考虑的三个配方(4.8、4.82 和 4.9 BPE)中的每一个重复了此过程。表 16 和表 17 报告了用于最终候选筛选的准确性和冗长性结果(其中选定的候选检查点将接受更全面的评估,使用更广泛的基准测试)。额外 QAD 迭代的效果取决于配方。对于 4.8-BPE 配方,从 200 次迭代增加到 300 次迭代,既提高了准确性恢复,又改善了冗长性,将总 token 增加从 2.30% 降低到 0.75%。类似地,对于 4.9-BPE 配方,从 200 次迭代增加到 300 次迭代,提供了显著更好的恢复,并将冗长性变化从 1.79% 降低到与 BF16 持平。4.82-BPE 配方的表现不同:其 200 次迭代的检查点提供了最强的准确性恢复(几乎匹配 BF16 的中位准确性和冗长性),而 300 次迭代的检查点准确性较低,冗长性略低。总体而言,4.82-BPE 配方在 200 次迭代时提供了最强的准确性-冗长性权衡,并构成了最强的候选。我们还从其他两个配方中分别选择了最佳候选,用于最终阶段的全基准评估扫描。
We adjusted the QAD experiments based on the above observations for the final post-trained BF16 checkpoint. In particular, we reduced checkpointing interval to 100 iterations, used the 100% SFT blend only, and evaluated QAD checkpoints at 200, 300, and 400 iterations. We repeated this process for each of the three recipes considered previously: 4.8, 4.82, and 4.9 BPE. Tables 16 and 17 report the accuracy and verbosity results used for final candidate downselection (wherein selected candidate checkpoints undergo a more thorough evaluation with a broader set of benchmarks). The effect of additional QAD iterations is recipe-dependent. For the 4.8-BPE recipe, moving from 200 to 300 iterations improves both accuracy recovery and verbosity, reducing the aggregate token increase from 2.30% to 0.75%. Similarly, for the 4.9-BPE recipe, moving from 200 to 300 iterations provides substantially better recovery and reduces verbosity change from 1.79% to parity with BF16. The 4.82-BPE recipe behaves differently: its 200-iteration checkpoint provides the strongest accuracy recovery (almost matches median BF16 accuracy and verbosity), whereas the 300-iteration one is less accurate and slightly less verbose. Overall, the 4.82-BPE recipe at 200 iterations provides the strongest accuracy–verbosity tradeoff and constitutes the strongest candidate. We also selected the best candidate from each of the other two recipes for a final stage full benchmark evaluation sweep.
注意,尽管也评估了 400 次迭代的 QAD 检查点,但它们未列入表 16 和表 17,并且不影响候选筛选。我们的结果表明,主要的 QAD 收益在很大程度上到 300 次迭代时就饱和了。
Note that although the 400-iteration QAD checkpoints were also evaluated, they are omitted from Tables 16 and 17, and do not alter the candidate downselection. Our results in fact indicate that the principal QAD benefits largely saturate by 300 iterations.
作为研究的一部分,我们评估了 FP8 和 NVFP4 两种量化方案;在每种方案中,注意力机制层中的键缓存和值缓存分别被量化为 FP8 或 NVFP4。NVFP4 以 4 位 E2M1 值存储,每 16 个元素块使用一个 FP8 E4M3 缩放因子,因此每个缓存元素的有效存储成本为 4.5 位(不包括张量级缩放开销)。相对于 FP8 的每元素 8 位,这将理想 KV 缓存负载减少了 \(1-4.5/8=43.75\\%)。这是一个存储计算,而非运行时内存流量或延迟的实测减少;KV 缓存 BPE 与第 4.1 节中的全模型检查点 BPE 是分开的。
As part of our studies, we evaluate both FP8 and NVFP4 quantization cases; in each case both the key and value caches across attention layers are quantized in FP8 or NVFP4, respectively. NVFP4 stores 4-bit E2M1 values with one FP8 E4M3 scale per 16-element block, for an effective storage cost of 4.5 bits per cached element, excluding tensor-level scale overhead. Relative to FP8's 8 bits per element, this reduces the ideal KV cache payload by \(1-4.5/8=43.75\\%). This is a storage calculation, not a measured reduction in runtime memory traffic or latency; KV cache BPE is separate from the whole-model checkpoint BPE in Section 4.1.
我们的参考模型使用 BF16 权重和 FP8 KV 缓存。表 18 中的两种配置使用我们激进的 NVFP4 方案,在 200 次迭代下进行 QAD,并分别使用(均匀)NVFP4 或 FP8 KV 缓存进行评估。对于每个基准测试,我们对每种配置和参考模型的已完成运行取平均值,然后取 35 个配置与参考分数比率的中位数。NVFP4 KV 缓存配置的中位准确率恢复为 99.36%,发布的 FP8 KV 缓存配置为 99.52%(在相同的 35 个基准测试上)。
Our reference uses BF16 weights with an FP8 KV cache. The two configurations in Table 18 use our aggressive NVFP4 recipe with QAD at 200 iterations, evaluated with either (uniform) NVFP4 or FP8 KV cache. For each benchmark, we average completed runs within each configuration and the reference, then take the median of the 35 configuration-to-reference score ratios. Median accuracy recovery is 99.36% for the NVFP4 KV cache configuration and 99.52% for the released FP8 KV cache configuration (on the same 35 benchmarks).
NVFP4 KV 缓存实验使用原始 QAD 检查点,即具有固定的骨干 KV 缩放值,针对 NVFP4 KV 缓存部署(1/6);FP8 KV 缓存实验使用发布的量化检查点,该检查点不包含 KV 缩放,并针对 FP8 KV 缓存部署(vLLM 对 FP8 KV 缓存类型默认 KV 缩放为 1.0)。两者均使用 FP32 Mamba 卷积缓存和 FP16 SSM 缓存,并采用随机舍入。报告的相对于 BF16 权重参考的中位准确率恢复反映了完整的权重和缓存配置,而不仅仅是 KV 量化。结果表明,在所使用的特定量化方案中,总体准确率大致相当。尽管 NVFP4 KV 具有存储优势,但发布的量化 NVFP4 检查点出于运行时效率和软件就绪性的考虑,默认使用 FP8 KV 缓存。本比较未评估逐层混合 FP8 键/NVFP4 值(或反之)的配置。
The NVFP4 KV cache experiments use the original QAD checkpoint, i.e., with fixed backbone KV scale values targeting NVFP4 KV cache deployment (1/6); the FP8 KV cache experiments use the released quantized checkpoint that contains no KV scales and targets FP8 KV cache deployment (vLLM defaults to KV scales of 1.0 for FP8 KV cache type). Both use an FP32 Mamba convolution cache and FP16 SSM cache with stochastic rounding. The reported median accuracy recovery against the BF16-weight reference reflects the full weight-and-cache configuration, not KV quantization alone. The results show broadly comparable aggregate accuracy in the specific quantization recipe used. The released quantized NVFP4 checkpoint defaults to FP8 KV cache for runtime-efficiency and software-readiness considerations despite NVFP4 KV's storage advantage. Per-layer mixed FP8-key/NVFP4-value (or vice-versa) configurations are not evaluated in this comparison.
为理解量化对模型行为的影响,我们对三个组件进行了消融:全局注意力机制的投影、mamba 的投影以及 unembedding。我们将 W4A4 路由/共享专家与 FP8 KV cache 设为默认配置,以测试各投影对量化的敏感性。这揭示了一些有趣的发现:
To understand the impact of quantization on model behavior, we ablated three components: global attention's projection, mamba's projection, and the unembedding. We presumed W4A4 routed/shared experts and FP8 KV cache as the default configuration to test each projection's sensitivity to quantization. This reveals a few interesting findings:
MoE 专家权重对精度有主要影响,会在 FP4 量化的潜空间中引起 token 分布漂移。若弃答(abstention)的决策边界脆弱到足以被量化影响,这一点在推理失败时的失控循环生成中变得至关重要。任何注意力机制投影精度都无法恢复冗长膨胀(verbosity inflation)。经验上,提升 MoE 专家精度并在 RL rollout 上微调可缓解这一效应(附录 B.2)。
The MoE expert weights carry a primary impact on accuracy, causing a drift of token distribution in FP4-quantized latent space. This becomes critical for runaway loop generation in reasoning failures, if a decision boundary of abstention is fragile enough to be affected by quantization. No attention projection precision recovers the verbosity inflation. Empirically, promoting MoE expert precision and fine-tuning on RL rollouts can mitigate this effect (Appendix B.2).
由于长上下文检索主要由 KV cache 精度主导,注意力机制投影在多数情况下对精度影响极小。然而,表 19 显示,注意力分数稀释——即在众多 key 之间共享单一可表示的量化注意力分数水平——在 32k+ 上下文长度下进行消歧(如 MRCR-128k)时更为显著。q/k/v 投影不需要 BF16,但 per-tensor FP8 的动态范围限制了注意力 logit 的粒度。MXFP8 的 per-block scaling 有助于缓解质量退化。注意力机制的输出投影在结构上是暴露的,但对 FP8 量化具有容忍性(附录 B.3)。
Because long-context retrieval is dominated by KV cache precision, attention projection has minimal impact on accuracy in most cases. However, Table 19 shows that attention score dilution, sharing a single representable level of the quantized attention score among many keys, is more pronounced with disambiguation (e.g., MRCR-128k) in 32k+ context length. q/k/v projections do not require BF16, but per-tensor FP8's dynamic range constrains attention logit granularity. MXFP8's per-block scaling helps mitigate the quality degradation. Attention's output projection is structurally exposed but tolerant to FP8 quantization (Appendix B.3).
Mamba 输出投影对精度的影响也极小,但将其权重量化到 FP4 会膨胀生成长度。其输入激活则单独暴露:SSM 门发出接近零但从不精确为零的值,这些值在 FP4 激活量化下会被冲刷为零。旋转或更高精度可缓解激活中的这一效应,但若不进行权重重对齐,则无法解决冗长膨胀(附录 B.4)。
Mamba output projection also has a minimal impact on accuracy, but quantizing its weights to FP4 inflates generation length. Its input activations are separately exposed: the SSM gate emits near-zero yet never exactly zero values, which flush to zero under FP4 activation quantization. Rotation or higher precision can mitigate this effect in activation, but does not address the verbosity inflation without weight realignment (Appendix B.4).
即使采用 FP4 量化,unembedding 的 lm_head 权重也不会改变精度。然而,FP4 会放大停止与继续(stop-versus-continue)边界的扰动。为维持 token 生成的冗长度,lm_head 应保持 FP8 权重或更高精度(附录 B.5)。
The lm_head weights of unembedding do not change accuracy even with FP4 quantization. However, FP4 amplifies the perturbation of the stop-versus-continue margin. To maintain token generation verbosity, the lm_head should stay at FP8 weights or above (Appendix B.5).
为降低起草延迟,我们对 Nemotron 3.5 Super NVFP4 检查点的 MTP 模块进行了量化。这仅适用于 Nemotron 3.5 Super;Nemotron 3.5 Lightning 的 MTP 模块未量化。MTP 模块包含一个注意力机制层和一个 MoE 层。它没有自己的输出投影:发布的检查点存储单个 lm_head,在目标验证和 MTP 起草之间共享。性能分析表明,MoE 专家和 lm_head 占据了大部分起草延迟。因此,我们使用 NVFP4 W4A4 量化 MTP 专家。MTP 注意力机制权重保持 BF16,仅 K/V BMM 量化器使用 FP8。
To reduce drafting latency, we quantize the MTP module of the Nemotron 3.5 Super NVFP4 checkpoint. This applies only to Nemotron 3.5 Super; the MTP module of Nemotron 3.5 Lightning is not quantized. The MTP module contains an attention layer and an MoE layer. It does not carry its own output projection: the released checkpoint stores a single lm_head, shared between target verification and MTP drafting. Profiling shows that the MoE experts and the lm_head account for most of the drafting latency. We therefore quantize the MTP experts using NVFP4 W4A4. The MTP attention weights remain BF16, with only the K/V BMM quantizers at FP8.
隔离 MTP 量化的效果需要保持主干固定。在 NVFP4 主干与 BF16 MTP(起草长度 7 时 \(AL=4.612\))的对比中,量化 MTP 专家将 AL 降低至 \(AL=4.592\)(表 20)。
Isolating the effect of MTP quantization requires holding the backbone fixed. Against the NVFP4 backbone with a BF16 MTP (\(AL=4.612\) at draft length 7), quantizing the MTP experts reduces AL to \(AL=4.592\) (Table 20).
为了评估 MTP 和 lm_head 量化可能带来的加速,我们在单个 GB200 GPU 上,使用量化主干,模拟了起草长度为 \(5\) 时的起草和验证延迟。我们的模拟延迟预测显示,当 MTP 和 lm_head 均未量化时,起草占总延迟的 \(23.5\)%,起草延迟分为 \(49.6\)% 的 MTP 延迟和 \(51.4\)% 的 lm_head 延迟。因此,如果接受长度的下降不太大,使用量化来降低两个组件的延迟有利于减少总起草时间。
To assess the possible speedup from MTP and lm_head quantization, we simulated drafting and verification latencies at draft length \(5\), on a single GB200 GPU, with a quantized backbone. Our simulated latency projections show that when the MTP and lm_head are both unquantized, drafting accounts for \(23.5\%) of the total latency, and the drafting latency splits to \(49.6\%) MTP latency and \(51.4\%) lm_head latency. Therefore, using quantization to reduce the latency of both components is beneficial to reduce the total drafting time, if the drop in acceptance length is not too big.
根据我们的预测,量化 MTP 层预计可减少其 \(47\)% 的延迟,而将 lm_head 量化为 FP8 和 W4A16 精度预计分别减少其延迟 \(49\)% 和 \(71\)%。量化目标模型的 lm_head 最初被认为风险较高,因为它直接影响目标 logits,进而影响模型质量。作为后备方案,我们尝试了使用单独的目标和 MTP lm_head 精度的各种组合,并在 vLLM 0.30.0 中实现了支持,以这种混合配置服务模型。这允许目标 lm_head 保持 BF16,同时使用较低精度的 lm_head 进行起草。AL 结果和相对于使用 MTP 和 lm_head 的预期加速在表 20 中描述。
According to our projections, quantizing the MTP layer is expected to reduce \(47\%) of its latency, and quantizing the lm_head to FP8 and W4A16 precisions is expected to reduce its latency by \(49\%) and \(71\%), respectively. Quantizing the target model’s lm_head was initially considered higher risk because it directly affects the target logits and, consequently, model quality. As a fallback, we experimented with various combinations using separate target and MTP lm_head precisions, and implemented support in vLLM 0.30.0 for serving the model in this mixed configuration. This allows the target lm_head to remain in BF16 while using a lower-precision lm_head for drafting. The AL results and expected speedups over the using MTP and lm_head are described in Table 20.
最终选择了对目标 lm_head 进行 FP8 量化,因此单个 FP8 lm_head 同时服务于目标验证和 MTP 起草,无需单独的输出头。对于量化目标 lm_head 会降低质量的模型,单独精度支持仍然可用。
FP8 quantization of the target lm_head was ultimately selected, thus a single FP8 lm_head serves both target verification and MTP drafting, and no separate output head is required. The separate-precision support remains available for models where quantizing the target lm_head degrades quality.
MTP 无法使用与主干不同的 KV-cache 精度。因此,MTP 的 KV 精度跟随主干。我们通过实验验证,这不会降低 ALs,从而使该耦合成为一种结构性约束,而非质量上的权衡。
The MTP cannot use a different KV-cache precision from the backbone. MTP KV precision therefore follows the backbone. We empirically validated that this does not degrade ALs, making the coupling a structural constraint rather than a quality trade-off.
已发布的检查点是通过单独量化增强后的 MTP(第 3.7 节),再将其拼接进选定的量化主干而生成的,而非在单次流程中对两者联合量化。拼接过程会迁移 MTP 权重及其 scale,同时保持周边配置不变,因此源端与目标端必须在 MTP 张量集合上完全一致——包括名称、形状和 dtype。将 MTP 与主干联合量化可消除这一耦合,是未来发布的首选方向。
The released checkpoints were produced by quantizing the boosted MTP (Section 3.7) separately and splicing it into the selected quantized backbone, rather than co-quantizing both in a single pass. The splice transfers the MTP weights and their scales while leaving the surrounding configuration unchanged, so source and destination must agree exactly on the MTP tensor set — names, shapes and dtypes. Co-quantizing the MTP together with the backbone removes this coupling and is the preferred direction for future releases.
参照与 MTP 量化类似的流程,我们采用仅权重量化(weight-only PTQ)来提升 Nemotron Lightning 3.5 的 DSpark 模块与 Nemotron 3.5 Super 的 LiLiCorr 模块的草稿效率(见第 5.7 节)。实验表明,在内存受限场景下(例如并发度较低的边缘设备),并行草稿模型可从权重量化中显著受益,因为量化降低了加载草稿模型参数的内存开销。对于 Nemotron 3.5 Super,我们观察到将 LiLiCorr 的线性层量化为 NVFP4、同时将激活值保持为 BF16,可在 DGX Spark 上带来 11% 的吞吐提升,且不会明显牺牲接受长度,如表 28 所示。进一步量化 LM head 又带来了 5% 的吞吐提升。我们找到了在 DGX Spark 上于草稿模型准确率与整体吞吐两方面均表现良好的模型专属量化设置。更多细节见表 21。
Following a similar procedure to MTP quantization, we applied weight-only PTQ to improve the drafting efficiency of a DSpark module for Nemotron Lightning 3.5 and a LiLiCorr module for Nemotron 3.5 Super (see Section 5.7). Our experiments showed that in memory-bound scenarios, e.g., on edge devices with low concurrency, parallel drafters benefit greatly from weight quantization, as this reduces the memory overhead from loading drafter parameters. For Nemotron 3.5 Super, we observed that quantizing LiLiCorr's linear layers to NVFP4 while keeping the activations at BF16 resulted in an 11% throughput improvement on DGX Spark without meaningfully sacrificing acceptance lengths, as shown in Table 28. Further quantizing the LM head improved throughput by another 5%. We identified model-specific quantization settings that performed well both in terms of drafter accuracy and overall throughput on DGX Spark. Additional details are in Table 21.
我们使用为 Super 3.5 描述的 NeMo Gym 评估框架来评估 Nemotron 3.5 Lightning。Lightning 特定的设置可在 recipes 文件夹和可复现性教程 1818 18 https://github.com/NVIDIA-NeMo/Gym/blob/main/nemotron_recipes/lightning-3.5/reproducibility.md 中找到。为了评估,我们使用 FP8 KV 缓存和 FP32 Mamba SSM 缓存来服务 BF16 和 NVFP4 检查点。SSM 缓存保持为 FP32,因为在 DGX Spark 上对于低批量(< 4)用例,它并不是限制因素。在 W4A16 NVFP4 检查点中,路由专家、共享专家和语言模型输出头使用 4 位 NVFP4 权重和 16 位激活。Mamba 输入和输出投影使用 FP8 权重和激活,而注意力机制投影保持为 BF16。我们使用相同的 PTQ/QAD 流程选择 W4A16 NVFP4 配置,并在匹配的评估和服务设置下与 BF16 进行比较。表 22 比较了 BF16 和 W4A16 NVFP4 之间的基准分数 1919 19 评估结果可能因随机采样和(在适用情况下)评判器可变性而存在运行间差异。因此,分数可能与模型卡上报告的分数不同。,报告了中位准确率恢复率为 99.24%,以及跨基准的总计冗长变化比率为 \(-3.63\)%。
We evaluate Nemotron 3.5 Lightning using the NeMo Gym evaluation framework described for Super 3.5. Lightning-specific settings are available in the recipes folder and reproducibility tutorial1818 18 https://github.com/NVIDIA-NeMo/Gym/blob/main/nemotron_recipes/lightning-3.5/reproducibility.md. For evaluation, we serve both the BF16 and NVFP4 checkpoints with FP8 KV cache and FP32 Mamba SSM cache. SSM cache is kept in FP32 as it was not a limiter on DGX Spark for low batch (< 4) use cases. In the W4A16 NVFP4 checkpoint, the routed experts, shared experts, and language-model output head use 4-bit NVFP4 weights with 16-bit activations. The Mamba input and output projections use FP8 weights and activations, while attention projections remain in BF16. We select the W4A16 NVFP4 configuration using the same PTQ/QAD procedure, and compare it with BF16 under matched evaluation and serving settings. Table 22 compares the benchmark scores1919 19 Evaluation results are subject to run-to-run variation due to stochastic sampling and, where applicable, judge variability. Scores may therefore differ from those reported on the model cards. between BF16 and W4A16 NVFP4, reporting a Median Accuracy Recovery of 99.24% and a Ratio of Totals Verbosity change of \(-3.63\%) across benchmarks.
分数、增量和恢复率显示时四舍五入到两位小数。计算使用未四舍五入的分数。CritPt 恢复率未定义,因为其 BF16 分数为零,并被排除在中位数之外。
Scores, deltas, and recoveries are rounded to two decimal places for display. Calculations use unrounded scores. CritPt recovery is undefined because its BF16 score is zero and is excluded from the median.
我们在涵盖智能体能力、推理与知识、多模态、对话能力与指令遵循、长上下文理解以及多语言性能的综合基准套件上评估了 Nemotron 3.5 Super。Nemotron 3.5 Super 和基线的所有评估结果均通过 Nemo Gym [20] 收集。为了可复现性,有关评估设置的更多细节可在 Nemo Gym 配方文件夹和相应的可复现性教程 [21] 中找到。
We evaluate Nemotron 3.5 Super on a comprehensive suite of benchmarks spanning agentic capabilities, reasoning and knowledge, multimodal, conversational ability and instruction following, long-context understanding, and multilingual performance. All evaluation results for Nemotron 3.5 Super and baselines were collected via Nemo Gym [20]. For reproducibility purposes, more details on the evaluation settings can be found in the Nemo Gym recipes folder and the corresponding reproducibility tutorial [21].
我们评估了由 PTQ 和 QAD 生成的候选 NVFP4 检查点,以选择与基线 BF16 相比最能保持准确性和冗长度的候选。为了公平比较,BF16 和 NVFP4 的评估和服务参数是相同的。
We evaluate the candidate NVFP4 checkpoints generated from PTQ and QAD to select the most accuracy-preserving and verbosity-maintaining candidate compared to the baseline BF16. The evaluation and serving parameters are common between the BF16 and NVFP4 for fair comparison.
我们使用两个指标来衡量模型智能:中位准确率恢复率和总冗长比率。前者通过平均 NVFP4 和 BF16 检查点在已完成运行上的准确率来计算,将 NVFP4 的平均值除以 BF16 的平均值,并报告这些比率在基准上的中位数。后者通过平均所有轨迹和运行中每条轨迹的输出令牌数来计算。然后,我们对每个检查点在基准上的这些平均值求和,并报告 NVFP4 总量相对于 BF16 总量的百分比变化。
We measure model intelligence with two metrics: Median Accuracy Recovery and Ratio of Totals Verbosity. The former is computed by averaging accuracy over completed runs for the NVFP4 and BF16 checkpoints, dividing the NVFP4 average by the BF16 average, and reporting the median of these ratios across benchmarks. The latter is computed by averaging output tokens per trajectory over all trajectories and runs. We then sum these averages across benchmarks for each checkpoint and report the percent change of the NVFP4 total relative to the BF16 total.
Terminal-Bench 显示,在智能体完成的 rollout 中,每条轨迹的平均输出令牌数变化为 +0.2%,在所有 rollout 的中位数上变化为 +0.8%,但在所有 rollout 的平均值上变化为 +15.7%,因为大约 4.5% 的未完成失控轨迹(每条约 70 万令牌)主导了平均值。为了避免扭曲冗长指标,我们排除了 Terminal-Bench 的未完成 rollout;如果包括在内,冗长变化(总比率)将增加到 \(2.99\)%。
Terminal-Bench shows a +0.2% change in mean output tokens per trajectory over agent-completed rollouts and +0.8% in the median over all rollouts, but +15.7% in the mean over all rollouts, because roughly 4.5% of non-completing runaway trajectories (about 700k tokens each) dominate the mean. To avoid skewing the verbosity metrics, we exclude the non-completed rollouts for Terminal-Bench; if included the Verbosity Change (Ratio of Totals) increases to \(2.99\%).
我们将每个基准运行两次并平均其分数,以考虑运行间的变异性,此外还有基准指定的重复次数。
We run each benchmark twice and average their scores to account for run-to-run variability, in addition to the number of repeats specified by the benchmark.
Nemotron 3.5 Super 面向交互式、长上下文智能体式工作负载而设计,此类请求通常共享系统指令、工具定义或检索到的上下文,并在多轮交互中不断扩展该上下文。因此,有效的服务性能不仅取决于峰值吞吐量或单请求延迟,还取决于在保持前缀复用的同时维持交互式的每用户生成速率。
Nemotron 3.5 Super was designed for interactive, long-context agentic workloads, where requests commonly share system instructions, tool definitions, or retrieved context and then extend that context over multiple turns. Effective serving performance therefore depends not only on peak throughput or single-request latency, but also on preserving prefix reuse while sustaining an interactive per-user generation rate.
为进行系统调优,我们采用一个 20 轮的代理工作负载,每轮包含 2,048 个输入 token 和 1,024 个输出 token;共享提示词与先前对话在后续轮次中可进行前缀复用。当某服务配置点报告的每用户生成速率至少为 50 tokens/s 时,我们称其可用。在这些可用配置点中,我们寻求最高的聚合输出吞吐量。所选定的工作点取决于平台与服务配置。
For system tuning, we use a 20-turn proxy workload with 2,048 input tokens and 1,024 output tokens per turn; the shared prompt and prior conversation are eligible for prefix reuse on subsequent turns. We call a serving point usable when its reported per-user generation rate is at least 50 tokens/s. Among these points, we seek the highest aggregate output throughput. The selected operating point depends on the platform and serving configuration.
这一场景特别强调服务栈的三个部分。第一,当推测解码改变执行恢复的位置时,前缀复用必须对混合 Mamba-Attention 模型仍然有效。第二,内核选择必须能以比单一全局策略更细的粒度进行控制,因为在平衡工作点上的最佳内核未必是延迟-吞吐量曲线两端各自胜出的内核。第三,同一模型必须能高效映射到具有不同内存容量、带宽和量化支持的加速器上。我们在 vLLM 中解决了这些需求,并在下文刻画其效果。
This regime places particular emphasis on three parts of the serving stack. First, prefix reuse must remain effective for a hybrid Mamba-Attention model when speculative decoding changes the position from which execution resumes. Second, kernel selection must be controllable at finer granularity than a single global policy because the best kernels at a balanced operating point need not be those that win at either endpoint of the latency-throughput curve. Third, the same model must map efficiently to accelerators with different memory capacities, bandwidths, and quantization support. We address these requirements in vLLM and characterize their effect below.
我们在 NVIDIA 数据中心与本地系统上验证了 Nemotron 3.5 Super,涵盖 A100、H100、B200、GB200 和 GB300 平台、DGX Spark 与 DGX Station,以及 RTX 级 GPU。这些平台跨越多个架构代际与部署规模。我们并非在所有平台上强加同一套服务配置,而是针对每个平台调优量化与投机解码。目标始终一致:在 20 轮代理工作负载上最大化总吞吐量,同时保持每用户至少 50 个输出 token/s。
We validated Nemotron 3.5 Super across NVIDIA datacenter and local systems, including A100, H100, B200, GB200 and GB300 platforms, DGX Spark and DGX Station, and RTX-class GPUs. These platforms span multiple architecture generations and deployment scales. Rather than impose one serving configuration everywhere, we tune quantization and speculative decoding for each platform. The objective remains the same: maximize aggregate throughput over the 20-turn proxy workload while sustaining at least 50 output tokens/s/user.
我们的内部性能模型将图 11 中的优势归因于 Nemotron 3.5 Super 对逐请求状态的处理方式、每个解码步产出的 token 数量,以及其专家之间交换的数据量。在每个解码步中,权重只读取一次并由整个批次共享,而每个请求都要读取并更新自己的 KV cache 与循环状态,这些状态还必须常驻 GPU 内存。因此,吞吐量取决于每个请求产生一个 token 的代价有多低,以及有多少请求分摊读取权重的成本。
Our internal performance model attributes the advantage in Figure 11 to how Nemotron 3.5 Super handles per-request state, how many tokens each decoding step yields, and how much data its experts exchange. In each decode step the weights are read once and shared by the whole batch, while every request reads and updates its own KV cache and recurrent state, which must also stay in GPU memory. Throughput therefore depends on how cheaply each request produces a token and on how many requests share the cost of reading the weights.
与 Qwen 模型类似,Nemotron 3.5 Super 也是混合架构,但它的注意力机制层更少、key-value 头更窄,并将其余的序列混合交给状态不随上下文增长的 Mamba-2 层,因此随着智能体式会话不断累积历史,其逐请求内存增长要慢得多。在这些长度下,每个 Qwen3.8 请求需要更多 GPU 内存,因此单张 GPU 上能容纳的请求更少,这限制了分摊每次权重读取的请求数量;其稀疏注意力机制减少了每步读取的 KV cache 量,但并未减少必须存储的量。Qwen3.5 存储的 KV cache 与 Qwen3.8 大致相当,且每一步都读取全部 KV cache,因此其每个请求在内存和带宽上的开销都高于 Nemotron 3.5 Super 的请求。
Like the Qwen models, Nemotron 3.5 Super is a hybrid, but it has fewer attention layers, with narrower key-value heads, and leaves the rest of sequence mixing to Mamba-2 layers whose state does not grow with context, so its per-request memory grows much more slowly as agentic sessions accumulate history. At these lengths each Qwen3.8 request needs more GPU memory, so fewer requests fit on a GPU, which limits how many share each weight read; its sparse attention reduces how much KV cache is read per step, but not how much must be stored. Qwen3.5 stores about as much KV cache as Qwen3.8 and reads all of it on every step, so each of its requests costs more in both memory and bandwidth than a Nemotron 3.5 Super request.
在概率式草稿采样下,MTP 头达到的平均接受长度高于两个 Qwen 模型(表 9),因此每一步每个请求产出更多 token;而 LatentMoE 交换的隐向量小于 Qwen 模型的隐藏状态,从而降低了每个 token 的专家并行通信量。Nemotron 3.5 Super 解码中一项较大的逐请求开销是更新 Mamba 状态,由于接受的 token 数量无法预先得知,投机解码会为一步中验证的每个 token 写出该状态;ReplaySSM(第 5.4 节)减少了这一流量及其所需的内存。
The MTP head reaches a higher average acceptance length than both Qwen models under probabilistic draft sampling (Table 9), so each step yields more tokens per request, and LatentMoE exchanges latent vectors that are smaller than the Qwen models' hidden states, which reduces the expert-parallel communication per token. A large per-request cost in Nemotron 3.5 Super's decoding is updating the Mamba state, which speculative decoding writes out for every token verified in a step because the number of accepted tokens is not known in advance; ReplaySSM (Section 5.4) reduces this traffic and the memory it requires.
对于纯 Attention 模型,前缀复用体现为按 token 存储的 KV 状态,这些状态存放在相对较小的缓存页中。混合模型还必须额外恢复计算恢复位置处的循环 Mamba 状态。该状态包含某一层的完整序列摘要,且每个缓存位置所占空间远大于一个 Attention KV 条目。因此,vLLM 按字节大小对齐 Attention 缓存组与 Mamba 缓存组的缓存页大小。在我们的目标 Mamba-2 模型上,一个 Mamba 缓存页所占的字节数至少相当于约 4K 个 token 的 Attention KV 缓存。因此,Mamba 状态通常以比 Attention KV 状态粗得多的 token 粒度保留。
For a pure Attention model, prefix reuse is represented by per-token KV state stored in relatively small cache pages. A hybrid model must additionally recover the recurrent Mamba state at the position where computation resumes. This state contains the complete sequence summary for a layer and is much larger per cached position than an Attention KV entry. vLLM therefore matches the cache-page sizes of the Attention and Mamba cache groups by byte size. On our target Mamba-2 model, one Mamba cache page occupies at least as many bytes as approximately 4K tokens of Attention KV cache. Consequently, Mamba states are normally retained on a much coarser token grid than Attention KV state.
当前缀缓存与推测解码结合时,这种粒度差异就变得重要。假设目标模型通过 token \(T\) 获得前缀缓存命中。为了草拟 token \(T+1\),草拟器需要目标模型在 token \(T\) 处的中间隐藏状态。前缀缓存包含 Attention KV 状态和一个检查点化的 Mamba 状态,但不包含该中间隐藏状态。vLLM 通过从最近存储的 Mamba 检查点恢复目标模型来重建该状态。如果 Mamba 块跨越 \(L_{\mathrm{page}}\) 个 token,最近的可恢复状态可能位于 \(T-L_{\mathrm{page}}\),从而迫使目标模型重放 token \(T-L_{\mathrm{page}}\) 到 \(T\),仅仅为了恢复 token \(T\) 处的隐藏状态。
This granularity matters when prefix caching is combined with speculative decoding. Suppose the target model obtains a prefix-cache hit through token \(T\). To draft token \(T+1\), the drafter needs the target model's intermediate hidden state for token \(T\). The prefix cache contains the Attention KV state and a checkpointed Mamba state, but not that intermediate hidden state. vLLM reconstructs it by resuming the target model from the most recent stored Mamba checkpoint. If the Mamba block spans \(L_{\mathrm{page}}\) tokens, the nearest resumable state may be at \(T-L_{\mathrm{page}}\), forcing the target model to replay tokens \(T-L_{\mathrm{page}}\) through \(T\) solely to recover the hidden state at token \(T\).
我们改进了这一路径:可选地在推测解码重放边界附近额外保留一个 Mamba 缓存页。例如,若某工作负载使用 16 token 的细粒度前缀匹配单元,则可将额外检查点放在 \(T-16\) 附近,而非 \(T-L_{\mathrm{page}}\)。这样,后续请求只需重放最后 16 个目标模型 token,而无需重放整个 Mamba 块。调度器与缓存管理器在共享前缀边界处联合选择并发布一个有效检查点,从而避免使用过期的循环状态,并以额外一个 Mamba 页为代价换取目标模型重计算量的大幅减少。
We improved this path by optionally retaining an additional Mamba cache page near the speculative-decoding replay boundary. For example, if a workload uses a 16-token fine-grained prefix-match unit, the additional checkpoint can be placed near \(T-16\) rather than \(T-L_{\mathrm{page}}\). A subsequent request then replays only the final 16 target-model tokens instead of an entire Mamba block. The scheduler and cache manager jointly select and publish a valid checkpoint at the shared prefix boundary, avoiding stale recurrent state while trading one additional Mamba page for substantially less target-model recomputation.
该问题在非 SSM 模型中不那么突出,因为其缓存页跨越的 token 数少得多,从前一页边界重放的代价很低。该问题对 Nemotron 3.5 Super 也比对循环状态更小的混合 SSM 模型更为重要:Mamba-2 更大的缓存页占用使 \(L_{\mathrm{page}}\) 更大,从而放大了更细粒度检查点所避免的工作量。由此形成的机制使前缀缓存与推测解码可组合:前缀缓存消除重复的预填充工作,而推测解码减少预填充之后目标模型的解码步数。
The issue is less pronounced for non-SSM models because their cache pages span far fewer tokens, so replaying from the preceding page boundary is inexpensive. It is also more important for Nemotron 3.5 Super than for hybrid SSM models with smaller recurrent states: Mamba-2's larger cache-page footprint makes \(L_{\mathrm{page}}\) larger and amplifies the work avoided by the finer checkpoint. The resulting mechanism makes prefix caching and speculative decoding composable: prefix caching removes repeated prefill work, while speculative decoding reduces the number of target-model decode steps after prefill.
在饱和状态下最大化吞吐量的内核,不一定是在中等批处理大小下能给出最佳端到端结果的内核。我们的目标恰恰位于这个中等区间:我们需要足够的批处理来获得高聚合吞吐量,同时保持至少 50 输出 tokens/s/用户。此外,混合精度检查点可能包含具有不同量化方案的线性层,其 GEMM 形状和最佳后端各不相同。因此,单一的进程级后端选择会将本应独立调优的决策耦合在一起。
The kernel that maximizes throughput at saturation is not necessarily the kernel that gives the best end-to-end result at an intermediate batch size. Our target lies in precisely this intermediate regime: we need enough batching to obtain high aggregate throughput while preserving at least 50 output tokens/s/user. Furthermore, a mixed-precision checkpoint may contain linear layers with different quantization schemes, whose GEMM shapes and best backends differ. A single process-wide backend selection therefore couples decisions that should be tuned independently.
全局线性后端保持为默认值,而内核配置可以针对特定的量化方案覆盖它——例如,为 NVFP4 W4A16 层选择专用后端,同时将其他线性层保留在平台默认值上。这种更细粒度的控制让部署可以为每种量化方案和加速器选择最佳支持的 GEMM 实现,而无需维护单独的模型工件或强制每个量化层走相同的路径。它还使服务配置具有可移植性:Hopper、Blackwell 和紧凑型系统可以使用不同的后端映射,同时保持相同的模型和工作负载级目标。2323 23 https://github.com/vllm-project/vllm/pull/51204
The global linear backend remains the default, while a kernel configuration can override it for a specific quantization scheme - for example, selecting a specialized backend for NVFP4 W4A16 layers while leaving other linear layers on the platform default. This finer control lets a deployment choose the best-supported GEMM implementation for each quantization scheme and accelerator without maintaining separate model artifacts or forcing every quantized layer through the same path. It also makes the serving configuration portable: Hopper, Blackwell, and compact systems can use different backend maps while preserving the same model and workload-level objective. 2323 23 https://github.com/vllm-project/vllm/pull/51204
一项推理性能改进源自 Nemotron v3 Ultra 技术报告中的建议。在 Mamba-2 层的解码步骤中,对于非推测解码,我们有循环状态 \(S\) 以及每个 token 的输入键向量 \(k\)、值向量 \(v\)、衰减标量 \(\alpha\) 和查询向量 \(q\)。新的循环状态则为 \(S_{new} = \alpha S_{old} + k^{\top} v\),输出为 \(o = q S_{new}\)。对于推测解码,我们对 \(T\) 个输入 token 执行此操作,并写出每个可能的新状态,因为我们不知道将接受多少个 token。这导致解码极度受内存限制,并且由于内存容量有限,它还限制了推测解码的 batch_size * draft_length。
One inference performance improvement grows out of a suggestion in the Nemotron v3 Ultra Technical report. During a decoding step of the Mamba-2 layer, for non-speculative decoding we have the recurrent state \(S\) and per-token inputs key vector \(k\), value vector \(v\), decay scalar \(\alpha\), and query vector \(q\). The new recurrent state is then \(S_{new} = \alpha S_{old} + k^{\top} v\), and the output \(o = q S_{new}\). For speculative decoding we do this for the \(T\) incoming tokens’ worth of input, writing out each possible new state as we do not know how many tokens will be accepted. This results in decoding being extremely memory bound, and it also limits batch_size * draft_length for speculative decoding due to limited memory capacity.
我们此后实现了一种“检查点重放”方法。我们存储检查点状态 \(S_{ckpt}\),并将自检查点以来看到的 \(k_{i}, v_{i}, \alpha_{i}\) 存储在一个小缓冲区中。由于 \(S = (\prod \alpha_{i}) S_{ckpt} + \sum_{i} (\prod_{j>i+1} \alpha_{j}) k_{i}^{\top} v_{i}\),我们可以按需重建状态,并且除了 \(S_{ckpt}\) 中的量化误差外没有额外的量化误差。被拒绝的 token 通过简单地递减计数器来处理,新输入被追加到缓冲区。为了生成输出 token,我们可以重建状态,或者更便宜地直接生成输出 token,注意到(为说明忽略 \(\alpha\) 项)\(q S = q S_{ckpt} + \sum_{i} (q k_{i}^{\top}) v_{i}\)。当缓冲区填满时,我们通过重建状态并将其写出,来“刷新”或“折叠”它。对于推测解码,我们只包括历史中被接受的 token,该步骤的 token 作为新历史缓冲区的种子。
We have since implemented a "checkpointing replay" method. We store a checkpointed state \(S_{ckpt}\) and store the \(k_{i}, v_{i}, \alpha_{i}\) seen since the checkpoint in a small buffer. As \(S = (\prod \alpha_{i}) S_{ckpt} + \sum_{i} (\prod_{j>i+1} \alpha_{j}) k_{i}^{\top} v_{i}\) we can reconstruct the state on-demand, and with no quantization error beyond what’s in \(S_{ckpt}\). Rejected tokens are handled by simply decrementing a counter, and new inputs are appended to the buffer. To generate output tokens, we can either reconstruct the state or we can generate output tokens more cheaply directly, by noting that (ignoring the \(\alpha\) terms for exposition) \(q S = q S_{ckpt} + \sum_{i} (q k_{i}^{\top}) v_{i}\). When our buffer fills up, we ’flush’ or ’fold’ it by reconstructing the state and writing it out. For speculative decoding we only include the accepted tokens from our history, with that step’s tokens seeding the start of the new history buffer.
这将状态写入分摊到缓冲区长度上,将每步状态相关的内存流量从 \(T+1\) 减少到大约 \(1+1/L\)。它还让我们能够分摊与写入状态相关的任何计算开销,例如随机数生成、随机舍入或缩放计算。特别是对于推测解码,得到的公式看起来很像分块预填充,只是部分被移动了。与分块预填充一样,FLOP 成本上升,但我们可以将 CUDA 核心工作(即矩阵向量乘和外积)转移到 Tensor Cores,因此实际计算时间下降。总之,该方法:
This amortizes the state write over our buffer length, reducing our per-step state-related memory traffic from \(T+1\) to roughly \(1+1/L\). It also lets us amortize any computational overhead associated with writing the state, such as randomness generation, stochastic rounding, or scaling calculations. Especially for speculative decoding, the resulting formulae look a lot like chunked prefill, with the parts moved around. Like chunked prefill, the FLOP cost goes up but we get to move CUDA core work, namely matvecs and outer products, over to Tensor Cores, so actual compute time goes down. In summary, this method:
减少 IO 密集型操作的内存流量。
Reduces memory traffic on an IO-bound operation.
通过将工作转移到 Tensor Cores 来减少计算时间。
Reduces compute time by shifting work to Tensor Cores.
通过降低量化频率来减少量化误差。
Reduces quantization error by quantizing less often.
支持更大的批大小和推测长度。
Enables larger batch sizes and speculation lengths.
更多细节请参见 https://github.com/NVIDIA/TensorRT-LLM/pull/14203。我们注意到该技术由 Dao 实验室独立并行发现,他们称之为 ReplaySSM,如其博客文章(https://dao-lab.ai/blog/2026/replayssm/)所述;我们随后与他们合作完成了论文并将其引入 vLLM。
For more details, see https://github.com/NVIDIA/TensorRT-LLM/pull/14203. We note this technique was discovered concurrently and independently in the Dao lab, where it is called ReplaySSM, as described in their blog post (https://dao-lab.ai/blog/2026/replayssm/); we subsequently collaborated with them on the paper and on bringing it to vLLM.
该工作负载包含 20 轮,具有 51,200 个 token 的共享前缀,每轮输入 2,048 个 token,输出 1,024 个 token。
The workload comprises 20 turns with a 51,200-token shared prefix, 2,048-token input per turn, and 1,024-token output.
在对齐的 Mamba 缓存模式下,必须在缓存块边界处物化可重用的循环状态。调度路径可以通过在该边界结束预填充步骤,并为提示的剩余部分发出额外的模型前向来强制执行这一点。尽管这产生了所需的状态,但拆分重复了固定的前向传递开销,并增加了对注意力机制、MoE、路由和张量并行通信路径的另一次遍历。
In aligned Mamba cache mode, a reusable recurrent state must be materialized at a cache-block boundary. The scheduling path can enforce this by ending a prefill step at that boundary and issuing an additional model forward for the remainder of the prompt. Although this produces the required state, the split repeats fixed forward-pass overhead and adds another traversal of the Attention, MoE, routing, and tensor-parallel communication paths.
内部预填充检查点最初是在 vLLM 中为 Kimi K3 实现的。基于这项工作,我们在 PR #57329 中将共享检查点构建器和导出器适配到了 Mamba-2。Mamba 扫描已经在逻辑块边界处物化了中间循环状态。适配后的路径安排逻辑块在所需的缓存边界结束,在那里提取状态和卷积窗口,并将该检查点写入分页缓存,同时整个提示保持在一个模型前向中。因此,以前需要在检查点之后进行额外前向的提示现在可以在单次前向中处理,而不会牺牲可重用的 Mamba 状态。
Internal prefill checkpointing was initially implemented in vLLM for Kimi K3. Building on that work, we adapted the shared checkpoint builder and exporter to Mamba-2 in PR #57329. The Mamba scan already materializes intermediate recurrent states at logical chunk boundaries. The adapted path arranges for a logical chunk to end at the desired cache boundary, extracts the state and convolution window there, and writes that checkpoint into the paged cache while the full prompt remains in one model forward. A prompt that previously required an additional forward after the checkpoint can therefore be processed in a single forward without sacrificing the reusable Mamba state.
综上所述,这些更改针对的是 Nemotron 3.5 Super 部署最重要的场景:共享和增长的上下文、交互式解码速率,以及从吞吐量-延迟前沿的有用部分中选择的硬件特定配置。前缀感知的推测解码减少了重复的预填充和解码工作,按量化选择后端暴露了调整平衡工作点所需的内核选择,而内部 Mamba 检查点降低了创建工作负载所依赖的可重用状态的成本。
Taken together, these changes target the regime that matters most for Nemotron 3.5 Super deployment: shared and growing contexts, interactive decode rates, and hardware-specific configurations chosen from the useful portion of the throughput-latency frontier. Prefix-aware speculative decoding reduces both repeated prefill and decode work, per-quantization backend selection exposes the kernel choices needed to tune balanced operating points, and internal Mamba checkpoints reduce the cost of creating the reusable state on which the workload depends.
我们在 NVIDIA Vera Rubin(VR200)GPU 上,使用 NVIDIA Dynamo 和 vLLM 部署 Nemotron 3.5 Super VL NVFP4 检查点,每个模型实例使用一块 GPU(不使用张量并行)。工作负载回放来自 SWE-bench Pro 轨迹的多轮智能体式编程会话,具有长上下文和跨轮次的高前缀复用。投机解码使用 MTP 头,固定接受长度为 4.06,草稿长度为 5。可用点要求中位输出速度不低于 50 tokens/s/user,且中位首 token 时间(TTFT)低于 5 s。我们报告在满足该目标的最高吞吐负载下,每 GPU 的输出 tokens/s。
We serve the Nemotron 3.5 Super VL NVFP4 checkpoint with NVIDIA Dynamo and vLLM on NVIDIA Vera Rubin (VR200) GPUs, with one GPU per model instance (no tensor parallelism). The workload replays multi-turn agentic coding sessions from SWE-bench Pro traces, with long contexts and high prefix reuse across turns. Speculative decoding uses the MTP head with a fixed acceptance length of 4.06 and draft length 5. A usable point keeps the median output speed at or above 50 tokens/s/user and the median time to first token (TTFT) below 5 s. We report output tokens/s per GPU at the highest-throughput load that meets this target.
Prefill 读取提示并构建 KV cache,即先前 token 的已存储注意力状态。随后 Decode 生成输出 token。我们比较两种部署,每种各使用 11 块 GPU。聚合式部署运行 11 个独立实例,每个实例同时执行 prefill 和 decode。一个感知 KV cache 的路由器将每个请求发送到一个实例。分离式部署使用 7 块 GPU 仅用于 prefill,4 块 GPU 仅用于 decode。在 prefill 之后,KV cache 从 prefill GPU 迁移到 decode GPU。
Prefill reads the prompt and builds the KV cache, the stored attention state of earlier tokens. Decode then generates the output tokens. We compare two deployments on 11 GPUs each. The aggregated deployment runs 11 independent instances that each do prefill and decode. A KV-cache-aware router sends each request to an instance. The disaggregated deployment uses 7 GPUs for prefill only and 4 GPUs for decode only. After prefill, the KV cache moves from the prefill GPU to the decode GPU.
分离式服务比聚合式服务每 GPU 的输出 tokens/s 高 10.3%(表 25)。在其最佳点,每用户的中位输出速度也高出 61%。中位 TTFT 从 0.49 s 增加到 1.12 s,但仍远低于 5 s 的目标。Prefill 和 decode 不再共享一块 GPU。因此,decode GPU 可以运行更大的 decode 批次,而无需在 decode 步骤之间插入长 prefill。
Disaggregated serving gives 10.3% more output tokens/s per GPU than aggregated serving (Table 25). At its best point, the median output speed per user is also 61% higher. The median TTFT increases from 0.49 s to 1.12 s, but stays well below the 5 s target. Prefill and decode no longer share a GPU. As a result, the decode GPUs run larger decode batches without long prefills between decode steps.
两种部署均使用以下引擎设置。权重使用 NVFP4,KV cache 使用 FP8。Mamba 状态缓存使用 FP16 并采用随机舍入,如第 4 节所述。编译器将每个注意力机制操作与其输出的量化融合。引擎为完整和分段模型步骤捕获 CUDA 图。在 decode 注意力机制中,一波 GPU 工作每个请求只允许对 KV cache 进行一到两次切分。多波 KV 切分随后进一步切分 KV cache,并在多个波中运行工作。我们将此选项添加到 FlashInfer 的 PR #6331(https://github.com/flashinfer-ai/flashinfer/pull/6331)和 vLLM 的 PR #61077(https://github.com/vllm-project/vllm/pull/61077)。
Both deployments use the engine settings that follow. The weights use NVFP4 and the KV cache uses FP8. The Mamba state cache uses FP16 with stochastic rounding, as in Section 4. The compiler fuses each attention operation with the quantization of its output. The engine captures CUDA graphs for full and piecewise model steps. In decode attention, one wave of GPU work can allow only one or two splits of the KV cache per request. The multi-wave KV split then splits the KV cache further and runs the work in several waves. We added this option to FlashInfer in PR #6331 (https://github.com/flashinfer-ai/flashinfer/pull/6331) and to vLLM in PR #61077 (https://github.com/vllm-project/vllm/pull/61077).
MTP 头每步起草 5 个 token。前缀缓存将 Mamba 状态存储在 KV cache 块边界处,并在多个请求共享的前缀处存储额外检查点。一个 vLLM 选项(disable_eagle_block_drop)允许在开启投机解码时,缓存命中保留其最后一个完整块。每个聚合实例和每个 prefill GPU 还在主机(CPU)内存中拥有一个 240 GiB 的 KV cache 层级。该层级在 GPU 缓存逐出先前轮次的前缀后保留这些前缀。
The MTP head drafts 5 tokens per step. Prefix caching stores Mamba states at KV-cache block boundaries, plus extra checkpoints at prefixes that several requests share. A vLLM option (disable_eagle_block_drop) lets a cache hit keep its last full block when speculative decoding is on. Each aggregated instance and each prefill GPU also has a 240 GiB KV cache tier in host (CPU) memory. This tier keeps the prefixes of earlier turns after the GPU cache evicts them.
每个 worker 都会发布 KV-cache 事件,因此路由器知道每个 worker 持有哪些缓存块。会话亲和性会把同一会话的所有轮次都发送到同一个 worker,因此后续轮次能在其缓存中找到前缀。在分离式部署中,NIXL 会把 KV cache 从 prefill GPU 迁移到 decode GPU。
Each worker publishes KV-cache events, so the router knows which cache blocks each worker holds. Session affinity sends all turns of one session to the same worker, so later turns find their prefix in its cache. In the disaggregated deployment, NIXL moves the KV cache from the prefill GPU to the decode GPU.
除了原生的 MTP 头(第 2.1 节和第 3.7 节),我们发布了 LiLiCorr,这是一个用于 Nemotron 3.5 Super 的独立草稿模型,面向低并发服务场景,在该场景中解码严重受限于内存带宽。此发布基于我们之前为 Nemotron 3.5 Lightning 所做的并行草稿模型工作,当时我们训练并发布了独立的 DFlash 和 DSpark 检查点。Nemotron 3.5 Lightning 的 DFlash 使用非因果的全序列分组查询注意力机制,而发布的 DSpark 使用因果分组查询注意力机制,并在每一层中采用滑动窗口注意力机制和每头注意力汇聚。将这些模型推向发布还需要 W4A16 后训练量化(第 4.8 节)以及针对 DSpark 马尔可夫头的 vLLM 优化。
Alongside the native MTP head (Sections 2.1 and 3.7), we release LiLiCorr, a standalone drafter for Nemotron 3.5 Super that targets low-concurrency serving, where decoding is strongly memory-bandwidth bound. This release builds on our earlier parallel-drafter work for Nemotron 3.5 Lightning, for which we trained and released separate DFlash and DSpark checkpoints. The Nemotron 3.5 Lightning DFlash uses non-causal, full-sequence grouped-query attention, whereas the released DSpark uses causal grouped-query attention with sliding-window attention in every layer and per-head attention sinks. Bringing these models to release also required W4A16 post-training quantization (Section 4.8) and vLLM optimizations for the DSpark Markov head.
在此针对的低并发场景中,验证一个草稿令牌块的成本与生成单个令牌相似,因此加速效果由接受长度和生成草稿的延迟共同决定。递归 MTP 头每次传递草稿一个令牌,因此其草稿延迟随草稿长度增长,而并行草稿模型在单次前向传递中提出所有草稿令牌,其延迟几乎与草稿长度无关。MTP 是模型检查点的一部分,不需要额外的权重,而 DFlash、DSpark 和 LiLiCorr 是针对此场景的独立草稿模型检查点。
In the low-concurrency regime targeted here, verifying a block of draft tokens costs similarly to generating a single token, so the speedup is determined jointly by the acceptance length and the latency of producing the draft. The recursive MTP head drafts one token per pass, so its drafting latency grows with the draft length, while parallel drafters propose all draft tokens in a single forward pass whose latency is nearly independent of the draft length. MTP is part of the model checkpoint and requires no additional weights, whereas DFlash, DSpark, and LiLiCorr are separate drafter checkpoints aimed at this regime.
并行草稿模型在每个块位置(以下称为槽)上基于边缘分布进行训练,因此每个槽独立选择的令牌可能各自合理但联合起来不连贯。DSpark 通过半自回归设计解决了这个问题。它保持草稿模型主体并行,并添加一个轻量级顺序头,该头使用来自前一个槽所草稿令牌的低秩转移来偏置每个槽的 logits。在完整的 DSpark 设计中,一个单独的置信度头估计每个草稿前缀被接受的概率,允许服务引擎根据请求和系统负载调整验证长度。Lightning 3.5 版本使用了轻量级马尔可夫校正但省略了此置信度头。LiLiCorr 则使用轻量级单次传递相关头来评分跨槽的兼容性。
A parallel drafter is trained on the marginal distribution at each block position, henceforth a slot, so tokens selected independently at each slot can be individually plausible yet jointly incoherent. DSpark addresses this with a semi-autoregressive design. It keeps the drafter body parallel and adds a lightweight sequential head that biases the logits at each slot using a low-rank transition from the token drafted at the previous slot. In the full DSpark design, a separate confidence head estimates the probability that each draft prefix will be accepted, allowing the serving engine to adapt the verification length to the request and system load. The Lightning 3.5 release used the lightweight Markov correction but omitted this confidence head. LiLiCorr instead uses a lightweight, single-pass correlation head to score compatibility across slots.
在 \(B-1\) 个草稿槽位中的每一个,LiLiCorr 保留 drafter body 边缘分布中概率最高的 \(K\) 个 token 作为候选,并用一个小型 Transformer 联合处理由此得到的 \((B-1)\times K\) 候选格,使每个候选都在块内所有其他候选的上下文中得到表示。对于槽位 \(s\) 处的第 \(k\) 个候选(\(k=1,\dots,K\),\(s=1,\dots,B-1\)),该 head 输出一个单位范数的 out 向量 \(\mathbf{o}_{s,k}\) 和一个 in 向量 \(\mathbf{i}_{s,k}\)。槽位 \(0\) 保存目标模型最后提交的 token。由于该 token 已经固定,它是槽位 \(0\) 处唯一的候选,并携带单个 out 向量 \(\mathbf{o}_{0,1}\)。槽位 \(s\) 处的候选 \(k\) 与槽位 \(s+1\) 处的候选 \(k^{\prime}\) 的兼容性,定义为前者的 out 向量与后者的 in 向量之间的余弦相似度:
At each of the \(B-1\) drafted slots, LiLiCorr retains the \(K\) most probable tokens of the drafter body's marginal as candidates and processes the resulting \((B-1)\times K\) candidate lattice jointly with a small Transformer, so that each candidate is represented in the context of all others in the block. For the \(k\)-th candidate at slot \(s\), with \(k=1,\dots,K\) and \(s=1,\dots,B-1\), the head emits a unit-norm out vector \(\mathbf{o}_{s,k}\) and in vector \(\mathbf{i}_{s,k}\). Slot \(0\) holds the last token committed by the target. Since this token is already fixed, it is the only candidate at slot \(0\) and carries a single out vector \(\mathbf{o}_{0,1}\). The compatibility of candidate \(k\) at slot \(s\) with candidate \(k^{\prime}\) at slot \(s+1\) is the cosine similarity between the out vector of the former and the in vector of the latter:
其中 \(\sigma\) 是固定尺度。记 \(\hat{c}_{s}\) 为槽位 \(s\) 处提交的候选索引,且 \(\hat{c}_{0}=1\),则块从左到右依次提交,每个槽位选择与其已提交前驱最兼容的候选:
where \(\sigma\) is a fixed scale. Writing \(\hat{c}_{s}\) for the index of the candidate committed at slot \(s\), with \(\hat{c}_{0}=1\), the block is committed from left to right, each slot selecting the candidate most compatible with its committed predecessor:
在概率式草稿采样下,每个槽位改为按请求温度从 head 在其候选上的分布中抽取。所有向量均来自 head 的一次前向,且式 (4) 中的成对得分可归约为批量矩阵乘积,因此只有游走过程是串行的,且无需进一步的网络评估。验证过程保持不变,因此目标模型的输出分布被精确保持。
Under probabilistic draft sampling, each slot is instead drawn from the head's distribution over its candidates at the request's temperature. All vectors come from one pass of the head and the pairwise scores in Eq. (4) reduce to batched matrix products, so only the walk is sequential, and it requires no further network evaluation. Verification is unchanged, so the output distribution of the target is preserved exactly.
我们在 NVIDIA Model Optimizer(ModelOpt)中联合训练 drafter body 与 LiLiCorr head,采用适合此规模目标的流式配置。目标语言模型的副本由 vLLM 以 BF16 提供,每个副本使用四路张量并行,并将五个捕获的目标层隐藏状态连同最终隐藏状态流式传输到 \(64\) 块训练 GPU。下方目标所需的下一词分布,在训练器上通过目标模型自身的 LM head 从最终隐藏状态重建,因此目标主体从不在训练器 rank 上复制。drafter 使用 \(B=8\) 个位置的块,即 \(7\) 个草稿槽位。drafter body 的五层中每一层都将注意力机制与 MLP 子层包裹在受 DFlash2 启发的分组动态卷积中,使相邻槽位能够交换局部信息,同时块保持并行。LiLiCorr head 保持原工作的配置。所得 drafter 具有 \(1.13\) B 参数,其中 \(37\) M 属于该 head。
We train the drafter body and the LiLiCorr head jointly in NVIDIA Model Optimizer (ModelOpt), in a streaming configuration suited to a target of this size. Replicas of the target's language model are served in BF16 by vLLM, each with tensor parallelism of four, and stream the hidden states of five captured target layers, together with the final hidden state, to \(64\) trainer GPUs. The target's next-token distribution, which the objective below requires, is reconstructed on the trainer from the final hidden state through the target's own LM head, so the target body is never replicated on the trainer ranks. The drafter uses a block of \(B=8\) positions, i.e., \(7\) drafted slots. Each of the five layers of the drafter body wraps the attention and MLP sublayers in a grouped dynamic convolution inspired by DFlash2, which lets neighboring slots exchange local information while the block remains parallel. The LiLiCorr head keeps the configuration of the original work. The resulting drafter has \(1.13\) B parameters, of which \(37\) M belong to the head.
原工作使用针对目标模型生成的训练响应 token 的交叉熵来训练 drafter body,并由 DFlash 的指数位置衰减加权。我们偏离这一配方,如 MTP boosting(第 3.7 节)那样,改为蒸馏目标模型 \(M\) 的完整下一词分布 \(p_{M}\)(该分布可从流式隐藏状态获得),同时保持相同的衰减。考虑一个从训练序列 \(x\) 的位置 \(t\) 开始的训练块,其中 \(x_{t}\) 位于槽位 \(0\),因此槽位 \(s\) 从前缀 \(x_{<t+s}\) 预测 token \(x_{t+s}\)。其损失为
The original work trains the drafter body with a cross-entropy against the tokens of the target-generated training responses, weighted by the exponential position decay of DFlash. We depart from this recipe and, as in MTP boosting (Section 3.7), instead distill the full next-token distribution \(p_{M}\) of the target model \(M\), which is available from the streamed hidden states, while keeping the same decay. Consider a training block that starts at position \(t\) of a training sequence \(x\), with \(x_{t}\) at slot \(0\), so that slot \(s\) predicts the token \(x_{t+s}\) from the prefix \(x_{<t+s}\). Its loss is
其中 \(\mathcal{V}\) 是词汇表,\(p_{\theta,s}\) 是参数为 \(\theta\) 的草稿模型主体在槽位 \(s\) 预测的、在 \(\mathcal{V}\) 上的边缘分布,\(\beta\) 设定衰减。损失在每个序列中采样起始位置的块上取平均。在不计常数的情况下,这是从 \(p_{M}\) 到 \(p_{\theta,s}\) 的前向 KL 散度。由于训练响应是在温度 \(1\) 下采样的,响应词元不一定是目标模型最可能的词元。完整分布同时监督目标模型最可能的词元(贪心验证测试的)和其分布的形状(采样验证测试的)。
where \(\mathcal{V}\) is the vocabulary, \(p_{\theta,s}\) is the marginal distribution over \(\mathcal{V}\) that the drafter body, with parameters \(\theta\), predicts at slot \(s\), and \(\beta\) sets the decay. The loss is averaged over blocks whose starting positions are sampled from each sequence. Up to a constant, this is the forward KL divergence from \(p_{M}\) to \(p_{\theta,s}\). Because the training responses are sampled at temperature \(1\), a response token need not be the target’s most likely one. The full distribution supervises both the target’s most likely token, which greedy verification tests, and the shape of its distribution, which sampled verification tests.
令 \(c_{s}\in\{1,\dots,K}\) 表示在槽位 \(s\) 选择的候选的索引,其中 \(c_{0}=1\)。头部在每个槽位定义了一个在 \(K\) 个候选上的分布,以先前槽位选择的候选为条件,
Let \(c_{s}\in\{1,\dots,K}\) denote the index of the candidate chosen at slot \(s\), with \(c_{0}=1\). The head defines, at each slot, a distribution over its \(K\) candidates conditioned on the candidate chosen at the previous slot,
其中 \(\psi\) 表示头部的参数。该分布也依赖于 \(\theta\),因为头部的输入包括候选在 \(p_{\theta,s}\) 下的对数概率以及草稿模型主体的隐藏状态。因此,头部的损失项也训练草稿模型主体。令 \(g_{s}\) 为与真实词元 \(x_{t+s}\) 相等的候选的索引(当它在 \(K\) 个候选之中时),其中 \(g_{0}=1\)。一个块的预言前缀是其最长的前导槽位序列 \(1,\dots,m\),这些槽位都包含真实词元,而 \(\mathcal{S}\) 收集了训练批次中所有块的预言前缀上的槽位。头部在真实前驱上接受教师强制,其交叉熵项与原始工作相同:
where \(\psi\) denotes the parameters of the head. The distribution depends on \(\theta\) as well, because the head’s inputs include the log-probabilities of the candidates under \(p_{\theta,s}\) and the drafter body’s hidden states. The head’s loss terms therefore train the drafter body as well. Let \(g_{s}\) be the index of the candidate equal to the ground-truth token \(x_{t+s}\), when it is among the \(K\) candidates, with \(g_{0}=1\). The oracle prefix of a block is its longest leading run of slots \(1,\dots,m\) that all contain the ground-truth token, and \(\mathcal{S}\) collects the slots on the oracle prefixes of all blocks in a training batch. The head is teacher-forced on the ground-truth predecessor, and its cross-entropy term is that of the original work:
原始工作用干扰项惩罚来补充 \(\mathcal{L}_{\mathrm{CE}}\),该惩罚惩罚头部分配给目标模型排名低于真实词元的候选的概率,惩罚力度与目标模型的对数概率差成比例。我们将其替换为校准项。令 \(q_{s,k}=p_{M}\!\left(v_{s,k}\mid x_{<t+s}\right)\) 为 \(v_{s,k}\) 的目标概率,即槽位 \(s\) 的第 \(k\) 个候选的词元,\(\bar{q}_{s,k}=q_{s,k}/\sum_{k^{\prime}}q_{s,k^{\prime}}\) 为其在 \(K\) 个候选上的重新归一化。校准项是从 \(\bar{q}\) 到头部每个槽位分布的交叉熵:
The original work complements \(\mathcal{L}_{\mathrm{CE}}\) with a distractor penalty, which penalizes the probability the head assigns to candidates that the target ranks below the ground truth, in proportion to the target’s log-probability gap. We replace it with a calibration term. Let \(q_{s,k}=p_{M}\!\left(v_{s,k}\mid x_{<t+s}\right)\) be the target probability of \(v_{s,k}\), the token of the \(k\)-th candidate at slot \(s\), and \(\bar{q}_{s,k}=q_{s,k}/\sum_{k^{\prime}}q_{s,k^{\prime}}\) its renormalization over the \(K\) candidates. The calibration term is the cross-entropy from \(\bar{q}\) to the head’s per-slot distribution:
干扰项惩罚被任何将所有概率放在目标模型排名等于或高于真实词元的候选上的分布所最小化。而校准项具有唯一的最小化器,即目标模型在每个槽位的候选上的重新归一化分布 \(\bar{q}_{s,\cdot}\)。这在概率性草稿采样下很重要,在 \(temp=1\) 时,每个草稿词元从头部每个槽位的分布中抽取。草稿模型主体和头部被联合训练于
The distractor penalty is minimized by any distribution that places all of its probability on candidates the target ranks at or above the ground truth. The calibration term instead has a unique minimizer, the target’s renormalized distribution \(\bar{q}_{s,\cdot}\) over the candidates of each slot. This matters under probabilistic draft sampling, which at \(temp=1\) draws each draft token from the head’s per-slot distribution. The drafter body and the head are trained jointly on
其中权重 \(\lambda_{\mathrm{CE}}\) 和 \(\lambda_{\mathrm{cal}}\) 在表 27 中给出。
其中权重 \(\lambda_{\mathrm{CE}}\) 和 \(\lambda_{\mathrm{cal}}\) 在表 27 中给出。
我们分两个阶段进行训练,这两个阶段都使用来自 12 个 Nemotron 后训练数据集的相同 6.32 M 条提示(列于表 26),但不同之处在于生成响应并作为流式目标的检查点。在第一阶段,这是后 SFT 的 Nemotron 3.5 Super 检查点。在第二阶段,是发布的 RL 后检查点,以便草稿模型适应其所服务模型的隐藏状态和输出分布。在第一阶段,每个样本的助手回合中随机选择一部分被重新生成,且每个样本以重新生成的回合结束。在第二阶段,所有助手回合都被重新生成。生成使用推理,温度 \(1.0\),top-\(p\) 为 \(0.95\),并限制为 \(8{,}192\) 个新 token。用户回合和工具结果逐字保留,且不执行任何工具。损失仅应用于最终的助手回合,该回合始终是重新生成的。第一阶段在 \(4{,}096\) 个 token 的窗口上训练,第二阶段在 \(8{,}192\) 个 token 的窗口上训练。经过滤后,第一阶段保留 \(5.67\) M 个对话,总计 \(22.6\) B 个 token,其中 \(14.9\) B 个落在其训练窗口内;第二阶段保留 \(18.2\) M 个生成样本中的 \(11.01\) M 个,总计 \(35.6\) B 个 token,其中 \(34.9\) B 个落在其训练窗口内。
我们分两个阶段进行训练,这两个阶段都使用来自 12 个 Nemotron 后训练数据集的相同 6.32 M 条提示(列于表 26),但不同之处在于生成响应并作为流式目标的检查点。在第一阶段,这是后 SFT 的 Nemotron 3.5 Super 检查点。在第二阶段,是发布的 RL 后检查点,以便草稿模型适应其所服务模型的隐藏状态和输出分布。在第一阶段,每个样本的助手回合中随机选择一部分被重新生成,且每个样本以重新生成的回合结束。在第二阶段,所有助手回合都被重新生成。生成使用推理,温度 \(1.0\),top-\(p\) 为 \(0.95\),并限制为 \(8{,}192\) 个新 token。用户回合和工具结果逐字保留,且不执行任何工具。损失仅应用于最终的助手回合,该回合始终是重新生成的。第一阶段在 \(4{,}096\) 个 token 的窗口上训练,第二阶段在 \(8{,}192\) 个 token 的窗口上训练。经过滤后,第一阶段保留 \(5.67\) M 个对话,总计 \(22.6\) B 个 token,其中 \(14.9\) B 个落在其训练窗口内;第二阶段保留 \(18.2\) M 个生成样本中的 \(11.01\) M 个,总计 \(35.6\) B 个 token,其中 \(34.9\) B 个落在其训练窗口内。
第一阶段从随机初始化开始训练 \(3\) 个 epoch,每个 epoch 有 \(88{,}539\) 步,总共 \(265{,}617\) 步,在 \(4{,}096\) 个 token 的窗口上进行。第二阶段从第一阶段结束时的全精度权重初始化,训练 \(500{,}000\) 步,约 \(2.9\) 个 epoch,每个 epoch 有 \(172{,}037\) 步,在 \(8{,}192\) 个 token 的窗口上进行,并使用新初始化的优化器和调度。目标模型在第一阶段使用四个副本提供服务,第二阶段使用八个副本。表 27 将配置与原始工作的配置进行了比较。
第一阶段从随机初始化开始训练 \(3\) 个 epoch,每个 epoch 有 \(88{,}539\) 步,总共 \(265{,}617\) 步,在 \(4{,}096\) 个 token 的窗口上进行。第二阶段从第一阶段结束时的全精度权重初始化,训练 \(500{,}000\) 步,约 \(2.9\) 个 epoch,每个 epoch 有 \(172{,}037\) 步,在 \(8{,}192\) 个 token 的窗口上进行,并使用新初始化的优化器和调度。目标模型在第一阶段使用四个副本提供服务,第二阶段使用八个副本。表 27 将配置与原始工作的配置进行了比较。
我们在 SPEED-Bench 的定性划分上进行评估,该划分包含 \(880\) 条提示,11 个类别中每个类别 \(80\) 条。草稿模型通过 vLLM 的 LiLiCorr 集成(27 27 27 https://github.com/vllm-project/vllm/pull/57934)提供服务,草稿长度为 \(7\),并发度为 \(64\)。我们报告每个样本的接受长度(AL),即每个验证步骤接受的 token 数,包括目标模型贡献的 token,先对每个请求求平均,再对请求求平均。\(temp=0\) 是贪婪解码,其中草稿模型执行公式 (5) 的游走。\(temp=1\),top-\(p\) 为 \(0.95\),使用公式 (7) 的概率草稿采样。对于两种设置,我们报告三个随机种子上的均值和标准差。我们使用 NVFP4 版本的 Nemotron 3.5 Super 模型进行验证,因为它对推理最有用,因此与草稿模型结合使用。
我们在 SPEED-Bench 的定性划分上进行评估,该划分包含 \(880\) 条提示,11 个类别中每个类别 \(80\) 条。草稿模型通过 vLLM 的 LiLiCorr 集成(27 27 27 https://github.com/vllm-project/vllm/pull/57934)提供服务,草稿长度为 \(7\),并发度为 \(64\)。我们报告每个样本的接受长度(AL),即每个验证步骤接受的 token 数,包括目标模型贡献的 token,先对每个请求求平均,再对请求求平均。\(temp=0\) 是贪婪解码,其中草稿模型执行公式 (5) 的游走。\(temp=1\),top-\(p\) 为 \(0.95\),使用公式 (7) 的概率草稿采样。对于两种设置,我们报告三个随机种子上的均值和标准差。我们使用 NVFP4 版本的 Nemotron 3.5 Super 模型进行验证,因为它对推理最有用,因此与草稿模型结合使用。
由于草稿模型在最多 \(8{,}192\) 个 token 的窗口上训练,其注意力机制无法外推到更长的上下文,并且在没有适应的情况下,其接受度会随着输入长度急剧下降。因此,我们在每个草稿模型层中对最近的 \(8{,}192\) 个位置使用滑动窗口注意力机制,使得查询-键距离永远不会超过训练中见到的距离。对于短于窗口的输入,这与全注意力机制相同。由于几乎所有 SPEED-Bench 定性样本都适合窗口,表 28 中的数字实际上是全注意力机制的数字。
由于草稿模型在最多 \(8{,}192\) 个 token 的窗口上训练,其注意力机制无法外推到更长的上下文,并且在没有适应的情况下,其接受度会随着输入长度急剧下降。因此,我们在每个草稿模型层中对最近的 \(8{,}192\) 个位置使用滑动窗口注意力机制,使得查询-键距离永远不会超过训练中见到的距离。对于短于窗口的输入,这与全注意力机制相同。由于几乎所有 SPEED-Bench 定性样本都适合窗口,表 28 中的数字实际上是全注意力机制的数字。
我们提出了 Nemotron 3.5 Super 与 Nemotron 3.5 Lightning,这是面向推理、长上下文理解与智能体式工作负载设计的高效混合专家(Mixture-of-Experts)Hybrid Mamba-Transformer 模型。通过大规模蒸馏、长上下文训练以及原生的多词元预测(Multi-Token Prediction,MTP),两个模型都将强大能力与高效推理结合在一起。Nemotron 3.5 Super 进一步引入了 Indic 继续预训练与智能体式软件工程中期训练,而涵盖监督微调、强化学习、专用教师训练与多教师同策略蒸馏的完整后训练流水线,将其转化为一个多模态智能体。Nemotron 3.5 Lightning 则采用更轻量的纯文本后训练流水线,针对高效部署进行了优化。
We presented Nemotron 3.5 Super and Nemotron 3.5 Lightning, efficient Mixture-of-Experts Hybrid Mamba-Transformer models designed for reasoning, long-context understanding, and agentic workloads. Through large-scale distillation, long-context training, and native Multi-Token Prediction (MTP), both models combine strong capabilities with efficient inference. Nemotron 3.5 Super further incorporates Indic continued pretraining and agentic software-engineering mid-training, while a comprehensive post-training pipeline spanning supervised fine-tuning, reinforcement learning, specialized teacher training, and multi-teacher on-policy distillation transforms it into a multimodal agent. Nemotron 3.5 Lightning follows a lighter text-only post-training pipeline optimized for efficient deployment.
我们的结果展示了在能力与效率两方面的提升。Indic 继续预训练将印度语基准提升最多 9.82 个百分点,而智能体式中期训练增强了长时程软件工程能力以及跨智能体框架(agent harness)的鲁棒性。最终的 Super 模型在 SWE-bench Pro 上达到 51.78%,尽管其激活参数约少 4.6 \(\times\),仍优于 Nemotron 3 Ultra。通过并行智能体执行与自我验证实现的测试时扩展(test-time scaling)进一步将 Terminal-Bench 2.1 从 62.7% 提升至 71.9%。在效率方面,后训练阶段的 MTP 增强在贪心解码下将推测解码的接受长度提升 10.1%,在概率性草稿采样下提升 21.8%;而 NVFP4 量化相对于 BF16 实现了 99.52% 的中位精度恢复。在 GB200 上,在所评估的交互式智能体式工作负载下,Super 的 serving 吞吐量达到 Qwen3.8 Flash-Next 的 1.9 \(\times\)。
Our results demonstrate improvements across capability and efficiency. Indic continued pretraining improves Indian-language benchmarks by up to 9.82 percentage points, while agentic mid-training strengthens long-horizon software engineering and robustness across agent harnesses. The final Super model achieves 51.78% on SWE-bench Pro, outperforming Nemotron 3 Ultra despite using approximately 4.6 \(\times\) fewer active parameters. Test-time scaling through parallel agent execution and self-verification further improves Terminal-Bench 2.1 from 62.7% to 71.9%. On the efficiency side, post-training MTP boosting improves speculative-decoding acceptance length by 10.1% under greedy decoding and 21.8% under probabilistic draft sampling, while NVFP4 quantization achieves 99.52% median accuracy recovery relative to BF16. On GB200, Super achieves 1.9 \(\times\) the serving throughput of Qwen3.8 Flash-Next under the evaluated interactive agentic workload.
综合来看,这些结果证明了联合优化预训练、后训练与推理,对于构建能力强且高效的智能体式模型所带来的收益。我们发布了模型检查点、数据集以及配套的配方与 cookbook,以促进开放智能体式 AI 系统的进一步研究与开发。
Together, these results demonstrate the benefits of jointly optimizing pretraining, post-training, and inference for capable and efficient agentic models. We release model checkpoints, datasets, and supporting recipes and cookbooks to enable further research and development of open agentic AI systems.
我们感谢以下人员对 NVIDIA Nemotron 3.5 Super 和 Lightning 所作出的宝贵贡献。
We thank the following people for their invaluable contributions to NVIDIA Nemotron 3.5 Super and Lightning.
Aastha Jhunjhunwala, Aayushi Ashokkumar Srivastava, Abhibha Gupta, Abhinav Garg, Abhinav Khattar, Achsah Wildish, Adam Rajfer, Adi Renduchintala, Adil Asif, Adriana Flores Miranda, Aileen Zaman, Aishwarya Chhabra, Akanksha Shukla, Akhiad Bercovich, Alejandra Rico, Aleksa Gordic, Aleksander Ficek, Alex Gronskiy, Alex Steiner, Alexander Bukharin, Alexandre Milesi, Ali Roshan Ghias, Ali Shehper, Ali Taghibakhshi, Alice Gatti, Alisa Liu, Alok Kumar, Alok Tripathy, Amala Sanjay Deshmukh, Aman Miryala, Aman Shanbhag, Ameya Sunil Mahabaleshwarkar, Ameya Velingker, Amir Klein, Amir Samani, Amit Zuker, Amnon Geifman, Anahita Bhiwandiwalla, Ananth Subramaniam, Andre Manoel, Andrea Santilli, Andrew Tao, Andrii Skliar, Anil Thomas, Anish Mahishi, Anna Shors, Anna Warno, Anuj Doshi, Anusha Kamath, Anwith Kiran, Ao Tang, Arham Mehta, Arnab Dhara, Arnav Komaragiri, Arnav Kundu, Arti Jain, Arun Rao, Arun Venkatesan, Arushi Goel, Asha Anoosheh, Ashish Vaswani, Ashwath Aithal, Asif Ahamed, Asma Kuriparambil Thekkumpate, Aswin Visva, Atefeh Sohrabizadeh, Avinash Vem, Ayush Dattagupta, Benedikt Schifferer, Benjamin Chislett, Bernard Nguyen, Besmira Nushi, Bilal Kartal, Bita Darvish Rouhani, Bobby Chen, Boris Ginsburg, Boxin Wang, Brandon Norick, Branislav Kisacanin, Brendan Johnson, Brian Dang, Brian Yu, Bryan Catanzaro, Byung-Kwan Lee, Carlo del Mundo, Chankyu Lee, Chantal Hwang, Charles Wang, Charlie Truong, Chen Cui, Cheng Wang, Cheng-Ping Hsieh, Chenhan Yu, Chenjie Luo, Chintan Patel, Chris Alexiuk, Chris Johnson, Chris Wing, Christian Munley, Coco Li, Collin McCarthy, Cynthia Gao, Dan Su, Dan Zhao, Dane Corneil, Danial Mohseni Taheri, Daniel Afrimi, Daniel Fatade, Daniel Lo, Daniel Machlab, Daniel Serebrenik, Daniele Molinari, Daniil Sorokin, Daria Gitman, Daria Levy, David Mosallanezhad, David Yu, Deepak Narayanan, Dev Churiwala, Devin O'Kelly, Dhruv Nathawani, Di Allan Wu, Dima Rekesh, Dina Yared, Dipendra Misra, Divyanshu Kakwani, Dmitry Belenko, Dominic Manno, Dongyang Fan, Dor Tsur, Duncan Riach, Dustin VanStee, Dylan Lu, Edgar Minasyan, Ehsan Hosseini-Asl, Eileen Long, Elena Lantz, Ellie Evans, Elliott Ning, Elvis Chen, Emanuel Scoullos, Eric Chung, Eric Harper, Eric Pham-Hung, Eric Tramel, Eric Yang, Erik Arakelyan, Evelina Bakhturina, Ewa Dobrowolska, Fadi Saady, Farzan Memarian, Fay Wang, Fei Jia, Felipe Soares, Felipe Vieira Frujeri, Feng Chen, Fengguang Lin, Ferenc Galko, Frankie Siino, Frederic Lafrance, Fuxiao Liu, Gantavya Bhatt, Garvit Kulshreshtha, George Armstrong, Giulio Lovisotto, Gomathy Venkata Krishnan, Gosia Lazuka, Grace Lam, Greg Heinrich, Greg Pauloski, Grzegorz Chlebus, Guilin Liu, Guo Chen, Guoming Zhang, Guyue Huang, Haifeng Qian, Hanrong Ye, Hao Gao, Hao Zhang, Haoqin Tu, Haran Kumar Shiv Kumar, Haribhau Hud, Harris Nover, Harrison Saturley-Hall, Harshil Kotamreddy, Hemil Desai, Henry Estela, Hexin Wang, Himanshu Singh, Hui Wang, Hung-Yueh Chiang, Igor Gitman, Ilia Karmanov, Ilya Loshchilov, Ingo Guehring, Itay Neeman, Ivan Moshkov, Jacky Kwok, Jakub Slowikowski, James Shen, Jane Polak Scowcroft, Jared Casper, Jatin Mitra, Jeff Willette, Jenna Diamond, Jenny Chen, Jesse Oliver, Jessica Medina, Jiacheng Xu, Jialin Song, Jian Hu, Jian Tu, Jian Zhang, Jiantao Jiao, Jiaqi Zeng, Jie-Fang Zhang, Jimmy Zhang, Jin Xu, Jingyu Xin, Jinhang Choi, Joey Conway, Joey Guman, John Kamalu, John Liu, John Roberts, Johnny Greco, Jonah Alben, Jonas Yang, Jonathan Cohen, Jonathan Li, Jonathan Raiman, Joseph Jennings, Juhi Mittal, Julia Gusak, Julien Veron Vialard, June Yang, Junkeun Yi, Jupinder Parmar, Ka Shun Shum, Kaiwen Shi, Kajal Jain, Kanishk Singla, Karan Sapra, Kari Briski, Kateryna Chumachenko, Katherine Huang, Katie Lyons, Keith Wyss, Keshav Santhanam, Keval Morabia, Kevin Shih, Kezhi Kong, Khanh Nguyen, Khushi Bhardwaj, Kiran Praveen, Konstantinos Krommydas, Krishna Puvvada, Krzysztof Pawelec, Kushal Shah, Li Ding, Lily Lee, Linda Liu, Linglin Jing, Luis Vega, Maarten Van Segbroeck, Mahan Fathi, Mahdi Nazemi, Maka Dong, Makesh Narsimhan Sreedhar, Maor Ashkenazi, Marc Cuevas, Marcin Chochowski, Marco Pavone, Markus Kliegl, Marta Stepniewska-Dziubinska, Martina Gonzalez Vilas, Martyna Patelka, Matan Rusanovsky, Matthieu Le, Matty Machczynski, Matvei Novikov, Mauricio Ferrato, Meenakshi Sharma, Mehrzad Samadi, Melissa Corpuz, Melissa Saelzer, Meng Xin, Mengru Wang, Mengxi Wu, Meriem Boubdir, Michael Boone, Michael Callahan, Michael Elad, Michael Lightstone, Michal Bien, Michal Zawalski, Miguel Martinez, Mikail Khona, Mike Chrzanowski, Mike Ranzinger, Mike Wojcikiewicz, Min-Hung Chen, Ming Kang, Mingjie Liu, Mingyuan Ma, Misha Nekorystnov, MJ Mikulski, Mohammad Qazim Bhat, Mohammad Shoeybi, Morgan Purcell, Mostofa Patwary, Nabin Mulepati, Nancy Agarwal, Narimane Hennouni, Nathan Sala, Nave Assaf, Nayeon Lee, Neal Vaidya, Negar Habibi, Netanel Haber, Nick Gupta, Nick Reamaroon, Nickson Quak, Nidhi Bhatia, Nik Spirin, Niket Agarwal, Nikolai Ludwig, Nima Tajbakhsh, Nithin Sunny, Oleksii Hrinchuk, Oleksii Kuchaiev, Olivier Delalleau, Omer Ullman Argov, Pablo Ribalta, Padmavathy Subramanian, Parth Mannan, Pavlo Molchanov, Pawel Morkisz, Peter Jin, Petr Lapukhov, Petro Junior Milan, Philip Monk, Pinky Xu, Piotr Januszewski, Piotr Laszkiewicz, Po-Han Huang, Praateek Mahajan, Pradeep Thalasta, Prakash Gurumurthy, Pranav Prashant Thombre, Prasoon Varshney, Pratyusha Maiti, Preeti Chadha, Pritam Biswas, Pritam Gundecha, Przemek Strzelczyk, Puhui Meng, Pulkit Kumar, Rabeeh Karimi Mahabadi, Rachel Oberman, Rajpreet Thethy, Ramani Akkati, Rameshwar Shivbhakta, Ran Zilberstein, Randy Gelhausen, Rasoul Shafipour, Raviraj Joshi, Ray Macalisang, Rayen Tian, Reka Kovacs, Renjie Pi, Resha Shetty, Ria C, Rima Shahbazyan, Rishabh Garg, Rita Fernandes Neves, Ritika Borkar, Ritu Gala, Riya Talwar, Riyad Islam, Robert Clark, Robert Kirby, Roger Waleffe, Rohan Joshi, Rohan Pillai, Rohit Jena, Rohit Watve, Roi Garnett, Roi Koren, Rongguang Ye, Russell J. Hewett, Ryan Prenger, Ryan Stewart, Ryota Egashira, Sadegh Mahdavi, Sai Nishanth Padala, Salika Dave, Sandhya Rani Narravula, Sandip Bhaskar, Sangkug Lym, Sanjeev Satheesh, Sarah Amirslani, Saran Vikas Murari, Saurabh Mishra, Saurabh Srivastava, Saurav Muralidharan, Sean Narenthiran, Sean Wang, Seokju Cho, Seong Hee Lee, Seonjin Na, Seonmyeong Bak, Seph Mard, Serge Panev, Sergei Kolchenko, Sergio Perez, Seth Henneman, Seth Poulos, Shahar Mor, Shantanu Acharya, Shaokun Zhang, Shaona Ghosh, Sharath Turuvekere Sreenivas, Shaun Kotek, Shaya Gharghabi, Sheng-Chieh Lin, Shengliang Xu, Shi Chen, Shihao Wang, Shilpa Amireddy, Shiqing Fan, Shiyang Chen, Shrimai Prabhumoye, Shriram Venugopal, Shuang Yu, Shubham Gandhi, Shubham Pachori, Shubham Toshniwal, Shuoyang Ding, Shwetha Krishnamurthy, Simeng Sun, Sivakumar Arayandi Thottakara, Somshubra Majumdar, Soumye Singhal, Sridurga Krithivasan, Sriharsha Niverty, Stephen Ge, Sugam Dipak Devare, Sukrit Rao, Sumeet Kumar Barua, Sungsoo Ha, Surya Nistala, Suseella Panguluri, Sweta Priyadarshi, Syeda Nahida Akter, Talor Abramovich, Tanay Varshney, Tandeep Singh Chandehoke, Teo Ene, Terry Kong, Thanh Do, Thinh Le, Tian Zhang, Tijmen Blankevoort, Tim Romanski, Tiyasa Mitra, Tom Balough, Tom Huang, Tomasz Grzegorzek, Tomer Asida, Tomer Bar Natan, Traian Rebedea, Trenton Starkey, Tugrul Konuk, Tuomas Rintamaki, Twinkle Vashishth, Tyler Kniess, Tyler Poon, Udi Karpas, Ushnish De, Utkarsh Vaidya, Vahid Noroozi, Vanshil Atul Shah, Venkat Srinivasan, Venkata Karri, Venmugil Elango, Vibhor Agrawal, Vibhu Jawa, Victor Cui, Victoria Edmonds, Vijay Korthikanti, Vikas Mehta, Vikranth Jeyakumar, Vinay Rao, Vineeth Kalluru Srinivas, Vitaly Kurin, Vitaly Lavrukhin, Wahidur Rahman, Wanli Jiang, Wasi Uddin Ahmad, Wataru Ishihara, Wei Du, Wei Ping, Wei-Ming Chen, Weiheng Chai, Weijia Chen, Wendy Quan, Wenliang Dai, Wenyan Cong, William Zhang, Wojciech Prazuch, Xiaowei Ren, Xin Yu, Xiwen Yu, Yan Wang, Yang Chen, Yang Yu, Yangyi Chen, Yaniv Galron, Yash Jain, Yashaswi Karnati, Ye Yu, Yejin Choi, Yemi Ibrahim, Yev Meyer, Yi Dong, Yi-Fu Wu, Yian Zhang, Yifan Zhang, Ying Lin, Yiyang Zhou, Yoav Miron, Yogendra Pandey, Yonatan Geifman, Yonggan Fu, Yoshi Suhara, Youngeun Kwon, Yu-Cheng Chou, Yu-Chiang Frank Wang, Yuanguo Kuang, Yuekai Zhang, Yugi Guvvala, Yuki Huang, Yunheng Zou, Yunsheng Liu, Zhenming Wang, Zhiding Yu, Zhiqi Li, Zhiyu Cheng, Zhongbo Zhu, Zhuolin Yang, Zichao Yue, Zihan Liu, Zijie Yan, Zirui Liu, Zoe Faul, Zuhair Ahmed
Aastha Jhunjhunwala, Aayushi Ashokkumar Srivastava, Abhibha Gupta, Abhinav Garg, Abhinav Khattar, Achsah Wildish, Adam Rajfer, Adi Renduchintala, Adil Asif, Adriana Flores Miranda, Aileen Zaman, Aishwarya Chhabra, Akanksha Shukla, Akhiad Bercovich, Alejandra Rico, Aleksa Gordic, Aleksander Ficek, Alex Gronskiy, Alex Steiner, Alexander Bukharin, Alexandre Milesi, Ali Roshan Ghias, Ali Shehper, Ali Taghibakhshi, Alice Gatti, Alisa Liu, Alok Kumar, Alok Tripathy, Amala Sanjay Deshmukh, Aman Miryala, Aman Shanbhag, Ameya Sunil Mahabaleshwarkar, Ameya Velingker, Amir Klein, Amir Samani, Amit Zuker, Amnon Geifman, Anahita Bhiwandiwalla, Ananth Subramaniam, Andre Manoel, Andrea Santilli, Andrew Tao, Andrii Skliar, Anil Thomas, Anish Mahishi, Anna Shors, Anna Warno, Anuj Doshi, Anusha Kamath, Anwith Kiran, Ao Tang, Arham Mehta, Arnab Dhara, Arnav Komaragiri, Arnav Kundu, Arti Jain, Arun Rao, Arun Venkatesan, Arushi Goel, Asha Anoosheh, Ashish Vaswani, Ashwath Aithal, Asif Ahamed, Asma Kuriparambil Thekkumpate, Aswin Visva, Atefeh Sohrabizadeh, Avinash Vem, Ayush Dattagupta, Benedikt Schifferer, Benjamin Chislett, Bernard Nguyen, Besmira Nushi, Bilal Kartal, Bita Darvish Rouhani, Bobby Chen, Boris Ginsburg, Boxin Wang, Brandon Norick, Branislav Kisacanin, Brendan Johnson, Brian Dang, Brian Yu, Bryan Catanzaro, Byung-Kwan Lee, Carlo del Mundo, Chankyu Lee, Chantal Hwang, Charles Wang, Charlie Truong, Chen Cui, Cheng Wang, Cheng-Ping Hsieh, Chenhan Yu, Chenjie Luo, Chintan Patel, Chris Alexiuk, Chris Johnson, Chris Wing, Christian Munley, Coco Li, Collin McCarthy, Cynthia Gao, Dan Su, Dan Zhao, Dane Corneil, Danial Mohseni Taheri, Daniel Afrimi, Daniel Fatade, Daniel Lo, Daniel Machlab, Daniel Serebrenik, Daniele Molinari, Daniil Sorokin, Daria Gitman, Daria Levy, David Mosallanezhad, David Yu, Deepak Narayanan, Dev Churiwala, Devin O'Kelly, Dhruv Nathawani, Di Allan Wu, Dima Rekesh, Dina Yared, Dipendra Misra, Divyanshu Kakwani, Dmitry Belenko, Dominic Manno, Dongyang Fan, Dor Tsur, Duncan Riach, Dustin VanStee, Dylan Lu, Edgar Minasyan, Ehsan Hosseini-Asl, Eileen Long, Elena Lantz, Ellie Evans, Elliott Ning, Elvis Chen, Emanuel Scoullos, Eric Chung, Eric Harper, Eric Pham-Hung, Eric Tramel, Eric Yang, Erik Arakelyan, Evelina Bakhturina, Ewa Dobrowolska, Fadi Saady, Farzan Memarian, Fay Wang, Fei Jia, Felipe Soares, Felipe Vieira Frujeri, Feng Chen, Fengguang Lin, Ferenc Galko, Frankie Siino, Frederic Lafrance, Fuxiao Liu, Gantavya Bhatt, Garvit Kulshreshtha, George Armstrong, Giulio Lovisotto, Gomathy Venkata Krishnan, Gosia Lazuka, Grace Lam, Greg Heinrich, Greg Pauloski, Grzegorz Chlebus, Guilin Liu, Guo Chen, Guoming Zhang, Guyue Huang, Haifeng Qian, Hanrong Ye, Hao Gao, Hao Zhang, Haoqin Tu, Haran Kumar Shiv Kumar, Haribhau Hud, Harris Nover, Harrison Saturley-Hall, Harshil Kotamreddy, Hemil Desai, Henry Estela, Hexin Wang, Himanshu Singh, Hui Wang, Hung-Yueh Chiang, Igor Gitman, Ilia Karmanov, Ilya Loshchilov, Ingo Guehring, Itay Neeman, Ivan Moshkov, Jacky Kwok, Jakub Slowikowski, James Shen, Jane Polak Scowcroft, Jared Casper, Jatin Mitra, Jeff Willette, Jenna Diamond, Jenny Chen, Jesse Oliver, Jessica Medina, Jiacheng Xu, Jialin Song, Jian Hu, Jian Tu, Jian Zhang, Jiantao Jiao, Jiaqi Zeng, Jie-Fang Zhang, Jimmy Zhang, Jin Xu, Jingyu Xin, Jinhang Choi, Joey Conway, Joey Guman, John Kamalu, John Liu, John Roberts, Johnny Greco, Jonah Alben, Jonas Yang, Jonathan Cohen, Jonathan Li, Jonathan Raiman, Joseph Jennings, Juhi Mittal, Julia Gusak, Julien Veron Vialard, June Yang, Junkeun Yi, Jupinder Parmar, Ka Shun Shum, Kaiwen Shi, Kajal Jain, Kanishk Singla, Karan Sapra, Kari Briski, Kateryna Chumachenko, Katherine Huang, Katie Lyons, Keith Wyss, Keshav Santhanam, Keval Morabia, Kevin Shih, Kezhi Kong, Khanh Nguyen, Khushi Bhardwaj, Kiran Praveen, Konstantinos Krommydas, Krishna Puvvada, Krzysztof Pawelec, Kushal Shah, Li Ding, Lily Lee, Linda Liu, Linglin Jing, Luis Vega, Maarten Van Segbroeck, Mahan Fathi, Mahdi Nazemi, Maka Dong, Makesh Narsimhan Sreedhar, Maor Ashkenazi, Marc Cuevas, Marcin Chochowski, Marco Pavone, Markus Kliegl, Marta Stepniewska-Dziubinska, Martina Gonzalez Vilas, Martyna Patelka, Matan Rusanovsky, Matthieu Le, Matty Machczynski, Matvei Novikov, Mauricio Ferrato, Meenakshi Sharma, Mehrzad Samadi, Melissa Corpuz, Melissa Saelzer, Meng Xin, Mengru Wang, Mengxi Wu, Meriem Boubdir, Michael Boone, Michael Callahan, Michael Elad, Michael Lightstone, Michal Bien, Michal Zawalski, Miguel Martinez, Mikail Khona, Mike Chrzanowski, Mike Ranzinger, Mike Wojcikiewicz, Min-Hung Chen, Ming Kang, Mingjie Liu, Mingyuan Ma, Misha Nekorystnov, MJ Mikulski, Mohammad Qazim Bhat, Mohammad Shoeybi, Morgan Purcell, Mostofa Patwary, Nabin Mulepati, Nancy Agarwal, Narimane Hennouni, Nathan Sala, Nave Assaf, Nayeon Lee, Neal Vaidya, Negar Habibi, Netanel Haber, Nick Gupta, Nick Reamaroon, Nickson Quak, Nidhi Bhatia, Nik Spirin, Niket Agarwal, Nikolai Ludwig, Nima Tajbakhsh, Nithin Sunny, Oleksii Hrinchuk, Oleksii Kuchaiev, Olivier Delalleau, Omer Ullman Argov, Pablo Ribalta, Padmavathy Subramanian, Parth Mannan, Pavlo Molchanov, Pawel Morkisz, Peter Jin, Petr Lapukhov, Petro Junior Milan, Philip Monk, Pinky Xu, Piotr Januszewski, Piotr Laszkiewicz, Po-Han Huang, Praateek Mahajan, Pradeep Thalasta, Prakash Gurumurthy, Pranav Prashant Thombre, Prasoon Varshney, Pratyusha Maiti, Preeti Chadha, Pritam Biswas, Pritam Gundecha, Przemek Strzelczyk, Puhui Meng, Pulkit Kumar, Rabeeh Karimi Mahabadi, Rachel Oberman, Rajpreet Thethy, Ramani Akkati, Rameshwar Shivbhakta, Ran Zilberstein, Randy Gelhausen, Rasoul Shafipour, Raviraj Joshi, Ray Macalisang, Rayen Tian, Reka Kovacs, Renjie Pi, Resha Shetty, Ria C, Rima Shahbazyan, Rishabh Garg, Rita Fernandes Neves, Ritika Borkar, Ritu Gala, Riya Talwar, Riyad Islam, Robert Clark, Robert Kirby, Roger Waleffe, Rohan Joshi, Rohan Pillai, Rohit Jena, Rohit Watve, Roi Garnett, Roi Koren, Rongguang Ye, Russell J. Hewett, Ryan Prenger, Ryan Stewart, Ryota Egashira, Sadegh Mahdavi, Sai Nishanth Padala, Salika Dave, Sandhya Rani Narravula, Sandip Bhaskar, Sangkug Lym, Sanjeev Satheesh, Sarah Amirslani, Saran Vikas Murari, Saurabh Mishra, Saurabh Srivastava, Saurav Muralidharan, Sean Narenthiran, Sean Wang, Seokju Cho, Seong Hee Lee, Seonjin Na, Seonmyeong Bak, Seph Mard, Serge Panev, Sergei Kolchenko, Sergio Perez, Seth Henneman, Seth Poulos, Shahar Mor, Shantanu Acharya, Shaokun Zhang, Shaona Ghosh, Sharath Turuvekere Sreenivas, Shaun Kotek, Shaya Gharghabi, Sheng-Chieh Lin, Shengliang Xu, Shi Chen, Shihao Wang, Shilpa Amireddy, Shiqing Fan, Shiyang Chen, Shrimai Prabhumoye, Shriram Venugopal, Shuang Yu, Shubham Gandhi, Shubham Pachori, Shubham Toshniwal, Shuoyang Ding, Shwetha Krishnamurthy, Simeng Sun, Sivakumar Arayandi Thottakara, Somshubra Majumdar, Soumye Singhal, Sridurga Krithivasan, Sriharsha Niverty, Stephen Ge, Sugam Dipak Devare, Sukrit Rao, Sumeet Kumar Barua, Sungsoo Ha, Surya Nistala, Suseella Panguluri, Sweta Priyadarshi, Syeda Nahida Akter, Talor Abramovich, Tanay Varshney, Tandeep Singh Chandehoke, Teo Ene, Terry Kong, Thanh Do, Thinh Le, Tian Zhang, Tijmen Blankevoort, Tim Romanski, Tiyasa Mitra, Tom Balough, Tom Huang, Tomasz Grzegorzek, Tomer Asida, Tomer Bar Natan, Traian Rebedea, Trenton Starkey, Tugrul Konuk, Tuomas Rintamaki, Twinkle Vashishth, Tyler Kniess, Tyler Poon, Udi Karpas, Ushnish De, Utkarsh Vaidya, Vahid Noroozi, Vanshil Atul Shah, Venkat Srinivasan, Venkata Karri, Venmugil Elango, Vibhor Agrawal, Vibhu Jawa, Victor Cui, Victoria Edmonds, Vijay Korthikanti, Vikas Mehta, Vikranth Jeyakumar, Vinay Rao, Vineeth Kalluru Srinivas, Vitaly Kurin, Vitaly Lavrukhin, Wahidur Rahman, Wanli Jiang, Wasi Uddin Ahmad, Wataru Ishihara, Wei Du, Wei Ping, Wei-Ming Chen, Weiheng Chai, Weijia Chen, Wendy Quan, Wenliang Dai, Wenyan Cong, William Zhang, Wojciech Prazuch, Xiaowei Ren, Xin Yu, Xiwen Yu, Yan Wang, Yang Chen, Yang Yu, Yangyi Chen, Yaniv Galron, Yash Jain, Yashaswi Karnati, Ye Yu, Yejin Choi, Yemi Ibrahim, Yev Meyer, Yi Dong, Yi-Fu Wu, Yian Zhang, Yifan Zhang, Ying Lin, Yiyang Zhou, Yoav Miron, Yogendra Pandey, Yonatan Geifman, Yonggan Fu, Yoshi Suhara, Youngeun Kwon, Yu-Cheng Chou, Yu-Chiang Frank Wang, Yuanguo Kuang, Yuekai Zhang, Yugi Guvvala, Yuki Huang, Yunheng Zou, Yunsheng Liu, Zhenming Wang, Zhiding Yu, Zhiqi Li, Zhiyu Cheng, Zhongbo Zhu, Zhuolin Yang, Zichao Yue, Zihan Liu, Zijie Yan, Zirui Liu, Zoe Faul, Zuhair Ahmed
推理长度本身并非缺陷;在智能体式工作负载中,长推理主要是在进行探查,例如运行代码、调用工具并读回结果。模型的弃答可能以两种机制失败:知识崩溃,会产生幻觉;以及推理崩溃,会产生终止失败,表现为语义上的失控循环行为。从定性上看,失控循环可被识别为失败的否定性证明:模型无法通过穷尽式调查来确证一个否定结果,却自信地重新打开一个已定的结论并循环往复。因此,一个无法终止的智能体会发出更多普通长度的调用,且每次调用的统计量保持稳定,而不是发出更长的调用。最终,该 rollout 的最后一段推理开始长度爆炸,或者截断率上升。
Reasoning length is not a defect; long reasoning in an agentic workload is mostly probing such as running code, invoking tools, and reading back the result. A model's abstention can fail by two mechanisms: knowledge collapse, which produces hallucination, and reasoning collapse, which produces termination failure, seen as semantic runaway looping behavior. Qualitatively, the runaway loop is recognizable as failed negative proof: the model cannot certify a negative result from exhaustive investigation, but confidently reopens a settled conclusion and cycles. Therefore, a non-terminating agent emits more calls of ordinary length, with per-call statistics steady, rather than longer calls. Eventually, the rollout's last reasoning starts exploding in length, or the truncation rate increases.
我们通过在事后剖析中阅读推理段落来识别失控循环,同时追踪模型在运行至上下文窗口上限时无法终止的失败。表 31 报告了两个可直接测量的量:截断,即至少包含一次触及输出上限的模型调用的 rollout 所占比例;以及每个 rollout 的输出长度。这两个量的组合是一个有用的早期预警指标。例如,tau3-banking 在 MOPD 之前有 20.9% 的截断率和每个 rollout 35,173 个输出 token,此时我们检测到 19.7% 的失控循环失败(153/776 个 rollout)。在 MOPD 之后,截断率降至 0.3%,每个 rollout 的输出为 13,669 个 token,此时仅捕获到一次失控循环失败。
We identify the runaway loop by reading reasoning passages in postmortem, while tracking the failure of a model to terminate as it runs to the context wall. Table 31 reports the two immediately measurable quantities: truncation, the fraction of rollouts containing at least one model call hitting an output limit, and output length per rollout. The combination of these two quantities is a useful early-warning indicator. For example, tau3-banking has 20.9% truncation and 35,173 output tokens per rollout before MOPD, where we detect 19.7% runaway loop failures (153/776 rollouts). After MOPD, truncation decreases to 0.3% with 13,669 output tokens per rollout, where only one runaway loop failure is captured.
一次评估框架(evaluation harness)更新导致了 ISL 差异。
a evaluation harness update resulted in ISL difference.
量化模型继承了基模型的失控循环行为,而量化可能使其更严重。我们通过在 pre-MOPD 模型中消融非注意力机制组件来分而治之地分析性能退化:
Quantized models inherit the runaway loop behavior from the base model, and quantization can make it worse. We divide and conquer the degradation by ablating non-attention components in pre-MOPD models:
路由/共享专家 NVFP4 \(\rightarrow\) FP8。在 tau3-banking 中,失控率从 28.2% \(\rightarrow\) 降至 17.9%,低于 BF16 参考的 19.7%;输出冗长度在 2.7 \(\sigma\) 下降低了 21.5%。这排除了专家权重无辜的可能性。
Routed/shared experts NVFP4 \(\rightarrow\) FP8. In tau3-banking, runaway dropped from 28.2% \(\rightarrow\) 17.9%, below the BF16 reference's 19.7%; output verbosity decreased by 21.5% at 2.7 \(\sigma\). This rules out expert weights being innocent.
专家激活 \(\rightarrow\) BF16。失控率变化 \(\approx 0\);输出冗长度在 5 个聚焦基准(mmlongbench-doc、scicode、tau3-banking、tau2-AA、AA-LCR)中的 5 个上均下降。这排除了激活路径作为终止失败载体的可能。
Expert activations \(\rightarrow\) BF16. Runaway moves by \(\approx 0\); output verbosity falls on 5 of 5 focused benchmarks (mmlongbench-doc, scicode, tau3-banking, tau2-AA, AA-LCR). This eliminates the activation path as the carrier of the termination failure.
lm_head 恢复为 BF16。未观察到终止行为的恢复。这排除了最终输出词生成作为失控循环来源的可能。
lm_head returned to BF16. No recovery of termination is captured. This eliminates the final output token generation as the source of the runaway loop.
W4A4 专家 GEMM 输出误差。相对于 BF16 的相对 L2 范数误差估计为 7.5–22.9%,在 40 个 MoE 层中有 39 个超过 10%,平均为 15%。这排除了“小扰动可以忽略”的说法。
W4A4 expert GEMM output error. The relative L2 norm error is estimated to be 7.5–22.9% against BF16, which is above 10% in 39 of 40 MoE layers, and 15% on average. This eliminates "the small perturbation can be ignored".
Mamba 输出投影 \(\rightarrow\) W4A16。在 tau3-banking 中,输出冗长度在 2.4 \(\sigma\) 处增加了 18.5%,而失控率和知识弃答均未受影响。
Mamba output projection \(\rightarrow\) W4A16. In tau3-banking, output verbosity increased by 18.5% at 2.4 \(\sigma\), while the runaway rate and knowledge abstention were both unaffected.
综合所有线索,我们推测罪魁祸首是专家权重的量化。将专家权重提升至 FP8 可缓解 pre-MOPD 模型上的失控循环,而对 post-MOPD 模型进行量化则成功恢复了冗长度,且 FP4 专家权重未出现失控循环回归。Mamba 的结果同样适用于 post-MOPD 检查点:与将输出投影保持在 W8A8 的量化方案相比,W4A16 在所有 6 项基准测试中均更为冗长。请参阅附录 B.4。注意,FP4 专家权重带来的扰动不会破坏信息,因此在所有实验中知识弃答均未受影响。在五次消融中,AA-OmniScience 非幻觉率仅在 \(0.991\times\) 至 \(1.022\times\) 之间波动,对应 \(67.8\%)和 \(69.9\%),而 BF16 为 \(68.4\%)。
By putting all the pieces together, we hypothesize the culprit is the quantization of the expert weights. Promoting expert weights to FP8 mitigates the runaway loop on pre-MOPD models, while post-MOPD model quantization successfully recovers the verbosity without runaway loop regression in FP4 expert weights. The Mamba result carries over to the post-MOPD checkpoints: against a quantization recipe keeping output projection at W8A8, W4A16 is more verbose on all 6 benchmark tests. Please refer to Appendix B.4. Note that a perturbation from FP4 expert weights does not destroy information, so knowledge abstention is not affected across all experiments. AA-OmniScience non-hallucination spans only \(0.991\times\) to \(1.022\times\) across the five ablations, which corresponds to \(67.8\%) and \(69.9\%) against BF16's \(68.4\%).
另一方面,MOPD 强化了停止的潜在支持,足以抵御 FP4 量化噪声。在 MOPD 下(这些 rollout 中仅 776 个里有 1 个出现循环),相同提示恢复至 \(0.081\) 和 \(0.077\),而 \(\beta\) 保持在较高的 \(0.16\),FP4 专家权重量化使其保持在 \(0.081\) 和 \(0.078\),仅报告了 3 例失控循环失败。表 33 显示,直接读取 unembedding 的决策证实了这一点:在实际发出 </think> 的位置,\(P(\text{stop})\) 为 \(0.99\),而其他位置为 \(10^{-9}\) 或更低,且 FP4 专家权重从未导致终止失败。FP4 专家影响扰动的方向。在 23 个 pre-MOPD rollout 中,有 17 个偏离停止;而在 MOPD 下,24 个中有 20 个朝向停止。在失控 rollout 本身,pre-MOPD 的峰值停止概率中位数为 \(0.043\),且推理块从未闭合;而 MOPD 达到中位数 \(0.729\),并在 10 个中有 6 个闭合了该块。GEMM 误差无法捕捉这一差异,因为它仅涵盖幅度,而不涵盖分布和偏差。
On the other hand, MOPD fortifies the latent support for stopping enough to withstand FP4 quantization noise. Under MOPD, which loops on only 1 of 776 of these rollouts, the same prompts recover to \(0.081\) and \(0.077\) while \(\beta\) stays at the elevated \(0.16\), and FP4 expert weight quantization leaves them at \(0.081\) and \(0.078\), where runaway loop failures are reported in 3 cases only. Table 33 shows that reading the unembedding's decision directly confirms this: at the position where </think> is actually emitted, \(P(\text{stop})\) is \(0.99\) against \(10^{-9}\) or less elsewhere, and FP4 expert weights never cost a termination. FP4 experts affect the direction of the perturbation. It moves away from stopping on 17 of 23 pre-MOPD rollouts, but toward stopping on 20 of 24 under MOPD. On the runaway rollouts themselves, pre-MOPD's peak stop probability has median \(0.043\) and the reasoning block never closes, while MOPD reaches median \(0.729\) and closes the block in 6 of 10. GEMM error could not capture this difference because it covers the magnitude only, not the distribution and bias.
一个 lm_head 将 </think> 置于顶部,分别为 \(0.384\) 和 \(0.871\),但采样未采纳它,循环继续。
a lm_head puts </think> on top, at \(0.384\) and \(0.871\), but sampling did not take it and the loop continued.
从稠密注意力机制的角度来看,长上下文检索对决策有三种不同的视角:
From the viewpoint of dense attention, long-context retrieval has three different perspectives on the decision:
区分:目标可以通过其内容识别。模型必须匹配候选者所说的内容,无论其位置在哪里。
Distinction: the target is identifiable by its content. The model must match on what a candidate says, wherever it sits.
消歧:候选者几乎相同,因此内容无法区分它们。模型必须按序数位置计数或索引出现次数。
Disambiguation: the candidates are near-identical, so content cannot separate them. The model must count or index occurrences by ordinal position.
聚合:没有单一目标;证据必须跨多个候选者组合。
Aggregation: there is no single target; evidence must be combined across many candidates.
在添加任何投影误差之前,注意力分数就已经分辨率不足。softmax 输出位于 \(0 \leq P \leq 1\) 区间内,浮点格式主要通过指数携带幅度。由于注意力机制输出 \(O = PV\),logit 扰动相当于对 \(v\) 向量进行位移。表 34 描述了在 12 个上下文(前缀 token 数从 10,532 到 63,641)上按头测量,其中 4,730 到 7,981 个键共置于注意力窗口内。这些键仅竞争 35 到 80 个不同的 e4m3 分数级别,导致每个级别上有 60 到 120 个键。
Attention scores are resolution-starved before any projection error is added. A softmax output lies in \(0 \leq P \leq 1\), where a floating-point format mostly carries the magnitude in its exponent. As attention delivers \(O = PV\), a logit perturbation is a bit shift on the \(v\) vector. Table 34 describes measuring per head across 12 contexts spanning 10,532 to 63,641 prefix tokens, where 4,730 to 7,981 keys are co-located in the attention window. These keys compete for only 35 to 80 distinct e4m3 score levels, putting 60 to 120 keys on each level.
假设 GraphWalks 的一个区分探针达到 361k 个观测上下文 token。由 logit 的指数性质,大多数逐位置候选在注意力分数上是平坦的,文档级聚合也是如此。注意力寻找的是与其他位置清晰分离的峰值分数,而需要区分与聚合的长上下文工作负载在 FP8 或 NVFP4 KV cache 精度下表现良好。不幸的是,如果候选在构造上近乎相同,除了位置之外没有任何东西能将它们区分开,那么精度本身就成了限制,正如 MRCR-128k 所展示的那样。
Suppose a distinction probe of GraphWalks, which reaches 361k tokens of observed context. From the exponential property of the logit, most per-position candidates are flat in attention scores. So is document-scale aggregation. While attention looks for a peak score, which is cleanly separated from other positions, a long-context workload requiring distinction and aggregation works well with FP8 or NVFP4 KV cache precision. Unfortunately, if candidates are near-identical by construction, so that nothing but position separates them, the precision becomes its limitation, as demonstrated by MRCR-128k.
尽管总注意力质量得以保留,逐层级的键集中会破坏量化中的分辨率。如表 19 所示,在 MRCR-128k 上,\(q/k/v\) 上 BF16 相对 FP8 的价值为 \(0.83\) pp,标准误为 \(0.71\) pp。这可能被解读为不显著的改进,因为 MXFP8 投影相对逐张量 FP8 获得 \(+0.99\) pp,使其位于 BF16 另一侧 \(0.16\) pp 处。在标准方差下,两个对比都未达到 \(2\sigma\)(= \(1.4\) pp)。同样,表 34 的最后一行可能暗示损害随上下文长度平滑增长。它并非线性增长。表 35 在三次重复下对 1,470 个 MRCR-128k 任务进行分层,以将检索与解码噪声分开。消融实验表明,损害在每个长度上都存在,但在 32k+ 处加剧:\(q/k/v\) FP8 在 32k 以下保持 \(95.3\)% 和 \(95.5\)%,而在 32k 以上为 \(91.4\)% 和 \(92.1\)%,出现 \(3.6\) pp 的断裂。相反,\(q/k/v\) MXFP8 投影在 32k 以上恢复到 \(95.3\)% 和 \(94.3\)%,与 \(q/k/v\) BF16 相似。在 \(q/k/v\) 上用 MXFP8 替代逐张量 FP8 只改变 scale 粒度,即每 32 个元素一个 E8M0 指数,而不是每个张量一个 float,这足以弥合差距。只有当共存的键容易混淆时,拥挤一个层级才会损失精度;而对于目标具有宽裕裕度的检索,无关上下文并不构成威胁。
Even though the total attention mass is preserved, per-level key concentration destroys the resolution in quantization. As shown in Table 19, the BF16-over-FP8 on \(q/k/v\) is worth \(0.83\) pp, with a standard error of \(0.71\) pp for MRCR-128k. This might be interpreted as an insignificant improvement, as MXFP8 projection gains \(+0.99\) pp against the per-tensor FP8, which places it \(0.16\) pp on the far side of BF16. Neither contrast clears \(2\sigma\) (= \(1.4\) pp) under the standard variance. Likewise, the last row of Table 34 might suggest damage grows smoothly with context length. It does not grow linearly. Table 35 stratifies 1,470 MRCR-128k tasks at three repetitions to separate the retrieval from decoding noise. The ablations show the damage is present at every length, but exacerbated at 32k+: \(q/k/v\) FP8 retains \(95.3\%) and \(95.5\%) below 32k, against \(91.4\%) and \(92.1\%) above 32k, a \(3.6\) pp break. On the contrary, \(q/k/v\) MXFP8 projection recovers \(95.3\%) and \(94.3\%) above 32k, similar to \(q/k/v\) BF16. Substituting MXFP8 for per-tensor FP8 on \(q/k/v\) changes only the scale granularity, with one E8M0 exponent per 32 elements instead of one float per tensor, which is enough to close the gap. Crowding a level only costs accuracy when the co-resident keys are confusable, and unrelated context does not threaten a retrieval whose target has a wide margin.
另一方面,表 36 描述了当候选数量从 2 增加到 8 时,BF16 从 0.800 崩溃到 0.378,并且这一排序在每个上下文区间内都成立。上下文也影响损害,但其影响并非单调递增:基线精度在 0–8k 与 8–32k 之间下降 \(13.4\) 至 \(29.5\) pp,然后饱和,因此在 32k 以上,候选数量是主导因素。这是索引预算耗尽的特征,而不是上下文窗口紧张的特征。
On the other hand, Table 36 describes the BF16 collapse from 0.800 to 0.378 as the candidate count rises from two to eight, and that ordering holds inside every context band. Context also affects the damage, but its impact is not monotonically increasing: the baseline accuracy steps down by \(13.4\) to \(29.5\) pp between 0–8k and 8–32k and then saturates, so above 32k the candidate count is the dominant factor. This is the signature of an exhausted indexing budget rather than a strained context window.
简而言之,损害在中间最严重,在两个候选时较小,因为裕度足够宽以吸收误差;在八个候选时又较小,因为参考已经崩溃,几乎没有可失去的。与附录 B.5 中的 unembedding 类似,量化会改变那些决策本已接近的结果。然而,注意力分数是分辨率机制的更强证据:一个仅仅让一切变得更难的机制不会放过最难的层级。从这一观察来看,报告单一数字会掩盖长上下文基准中的损害。为了正确评估量化损害,重要的是将测试用例与能力划分一起分层。
In short, the damage is worst in the middle, smaller at two candidates where the margins are wide enough to absorb the error, and smaller again at eight where the reference has already collapsed and little remains to lose. Similar to the unembedding in Appendix B.5, quantization moves outcomes where the decision was already close. However, attention score is stronger evidence for a resolution mechanism: one that merely made everything harder would not spare the hardest stratum. From this observation, reporting a single number hides the damage in long-context benchmarks. It is important to stratify test cases along with capability partition to assess quantization damage properly.
在 Nemotron 3.5 Lightning 模型上,当我们在保持权重和投影固定的情况下将 KV cache 从 FP8 移到 BF16 时,MRCR-128k 的精度恢复 \(+1.43\) pp(= \(2.4\sigma\));而当将投影从 FP8 移到 BF16 时,它恢复 \(+2.97\) pp(= \(2.8\sigma\))。在 Nemotron 3.5 Super VL 模型上,投影精度的影响从我们的消融实验中显现。虽然 KV cache 是损害载体,但高精度 \(q/k/v\) 投影并不能防止稀释;它买回了被转换破坏的部分裕度。表 37 在 AA-LCR 上变化注意力投影精度,这是一个平均服务 108k 上下文 token 的聚合基准。如前所述,聚合基准可以在 FP8 中识别重要候选,其中即使在 100 个任务、十六次重复下,逐张量 FP8 精度也可以接受。它还指出从全 FP8 到 NVFP4 投影的精度下降为 \(2.50\) pp,\(\mathrm{SE}=1.11\)。在 \(24\) 个 \(q/k/v\) 权重张量中,每 16 元素块最大值的散布中位数为 \(2.51\) binade,最差为 \(8.78\),而进入这些投影的激活跨度为 \(1.67\),相比之下 E4M3 块 scale 在下溢前编码 \(17.81\) binade。模型中没有块丢失其 scale,在最差张量上留下 \(9.03\) binade 未使用。逐块粒度并非在回收动态范围,而是恢复了逐张量 scale 已耗尽的精度余量。例如,在具有单一共享 scale 的融合 \(q/k/v\) FP8 投影下,键投影在八个注意力层中的六个上保持它,而值投影在另外两个上保持它。融合给 \(q\) 和 \(v\) 增加了约 32% 的相对权重误差,而 \(k\) 为 4%,这纯粹是舍入噪声。
On the Nemotron 3.5 Lightning model, when we move the KV cache from FP8 to BF16 while keeping weights and projections fixed, MRCR-128k's accuracy recovers \(+1.43\) pp (= \(2.4\sigma\)); while moving the projections from FP8 to BF16, it recovers \(+2.97\) pp (= \(2.8\sigma\)). On the Nemotron 3.5 Super VL model the impact of projection precision appears from our ablations. While the KV cache is the damage carrier, high-precision \(q/k/v\) projection does not prevent dilution; it buys back part of the margin destroyed by the cast. Table 37 varies attention projection precision on AA-LCR, an aggregation benchmark serving 108k tokens of context on average. As noted, aggregation benchmarks can identify important candidates in FP8, where even per-tensor FP8 precision can be acceptable over 100 tasks at sixteen repetitions. It also states an accuracy drop of \(2.50\) pp from all-FP8 to NVFP4 projections, with \(\mathrm{SE}=1.11\). Across the \(24\) \(q/k/v\) weight tensors, the spread of per-16-element block maxima is \(2.51\) binades at the median and \(8.78\) at worst, and the activations entering those projections span \(1.67\), which is against the \(17.81\) binades an E4M3 block scale encodes before it underflows. No block in the model loses its scale, leaving \(9.03\) binades unused on the worst tensor. Per-block granularity is not reclaiming dynamic range, but recovers precision headroom that a per-tensor scale had exhausted. For example, under a fused \(q/k/v\) FP8 projection with a single shared scale, the key projection holds it at six of eight attention layers and the value projection at the other two. Fusion adds roughly 32% relative weight error to \(q\) and \(v\) against 4% for \(k\), which is pure rounding noise.
本研究建议在 per-block 尺度粒度上对 \(q/k/v\) 投影采用 FP8 精度。然而,如果部署不执行消歧式检索,则没有其他工作负载会显现出显著差异。在 softmax 选定目标 \(v\) 向量之后,各注意力头的输出被送入输出投影。注意,在八个注意力层上,输出投影权重的分布范围中位数为 4.21 binade、最坏情况为 8.15 binade,而输出投影本身并不会遭遇 FP8 下溢。在激活侧,经校准的 per-tensor 输入尺度同样表现良好:在每一个注意力层上,它都处于最大头输出范数的 49–76% 之间,并在 \(68\times\) 的绝对幅值范围内保持该比例。反直觉的是,在单一 per-tensor 输入尺度下对各头输出进行拼接,并不会在 FP8 精度下产生基准级别的性能损失。我们断言,输出投影在结构上是暴露的,并且在 per-tensor FP8 下可测量地具有容忍度。同时值得指出的是,§4.3 中由基于梯度的逐层搜索所生成的混合精度检查点与这些发现一致:在 4.85 与 4.90 比特每参数两种预算下,AutoQuant 搜索均在八个注意力层中的三层上将 \(q/k/v\) 保留为 BF16,且两种预算下均为相同的三层;而 \(o\) 在 4.85 下于每个注意力层都保持 FP8,只有当预算放宽至 4.90 时才在三层上达到 BF16。
This study recommends FP8 precision on \(q/k/v\) projection at per-block scale granularity. If deployment does not perform disambiguation-style retrieval, however, no other workload would reveal a significant difference. After the softmax has chosen the target \(v\) vectors, the head outputs are passed to the output projection. Note that the spread of output projection weights is 4.21 binades in median and 8.15 in the worst case across eight attention layers, where output projection itself is not exposed to FP8 underflow. On the activation side, the calibrated per-tensor input scale is equally well-behaved: it sits at 49–76% of the largest head output norm at every attention layer, holding that fraction across a \(68\times\) range of absolute magnitude. Counterintuitively, the concatenation of head outputs under a single per-tensor input scale does not produce a benchmark-level penalty at FP8 precision. We claim that the output projection is structurally exposed and measurably tolerant in per-tensor FP8. In the meantime, it is worth noting that the mixed-precision checkpoints of §4.3, produced by a gradient-based per-layer search, follow these findings: the AutoQuant search leaves \(q/k/v\) in BF16 on three of the eight attention layers under both the 4.85 and 4.90 bit-per-parameter budgets, on the same three layers in each, while \(o\) stays FP8 on every attention layer under 4.85 and reaches BF16 on three layers only once the budget loosens to 4.90.
Mamba 输出投影与专家 fc2 的区别在于非线性输入的形态。专家 fc2 的输入由平方 ReLU 激活函数提供,在量化前其输入元素中有 90–99% 恰好为零,每个 FP4 块中仅有 1.05–3.32 个值存活。Mamba SSM 缓存将归一化的 SiLU 门传递给输出投影,该门提供接近零但从不恰好为零的值。因此,Mamba 输出投影的输入在每个 FP4 块中约有 16 个存活值,并且在 FP4 激活量化下,有 \(27.6\)% 的元素低于 \(\text{amax}_{\text{block}}/24\) 的舍入限制。这一张量分析阻止我们将 Mamba 输出投影用于 W4A4。应用旋转或更高的激活精度可以缓解这种冲刷现象。
The Mamba output projection differs from the expert fc2 in the input shape of the nonlinearity. Expert fc2 is fed by a squared-ReLU activation, where 90–99% of its input elements are exactly zero before quantization, and only 1.05–3.32 values survive per FP4 block. Mamba SSM cache passes a normalized SiLU gate to the output projection, which provides near-zero values but never exactly zero. Consequently, the input of Mamba output projection carries near 16 survivors per FP4 block, and \(27.6\)% of elements fall below the \(\text{amax}_{\text{block}}/24\) rounding limit under FP4 activation quantization. This tensor analysis stops us from taking the Mamba output projection to W4A4. Applying rotation or higher activation precision can mitigate the flush.
我们评估了 W4A16 下 FP4 权重量化与 FP8 W8A8 的影响对比。由于激活精度较高,这种对比直接揭示了权重量化的效果。表 38 显示了由 FP4 权重量化导致的输出冗长膨胀。所有六个基准测试都朝同一方向变化,其中 OSWorld 和 SciCode 的膨胀最为明显,分别为 \(1.27\)\(\times\) 和 \(1.16\)\(\times\)。注意,每个任务的准确率在两种方案下保持不变。为了解决这个问题,我们倾向于采用 §4.4 中描述的参数量化对齐方法。
We evaluated the impact of FP4 weight quantization with W4A16 against FP8 W8A8. Due to high precision on activation, this contrast reveals the weight quantization effect directly. Table 38 shows the output verbosity inflation caused by FP4 weight quantization. All six benchmarks move in the same direction, with OSWorld and SciCode carrying the clearest inflation at \(1.27\)\(\times\) and \(1.16\)\(\times\). Note that each task accuracy is unchanged across two recipes. To address this issue, we preferred parameter alignment as described in §4.4.
反嵌入层应保持 FP8 或更高精度。如表 39 所述,NVFP4 的 lm_head 也与 BF16 保持较高的 top-1 一致率。FP4 的问题在于停止与继续决策的失败,这导致输出冗长膨胀。
The unembedding should stay at FP8 or above. As described in Table 39, NVFP4 lm_head also holds high top-1 agreement with BF16. FP4's problem is a failure of the stop-versus-continue decision, which results in output verbosity inflation.
我们将该决策评估为单个 logit 比较,\(\text{margin}=h\cdot(v_{\text{stop}}-v_{\text{cont}})\。由于 </think> 是一个 token,两个反嵌入行共享的部分相互抵消。量化后,</think> 行与原模型的余弦相似度为 0.9961。该向量几乎未移动,然而 2.45% 的决策发生翻转,因为其他向量提供的余弦相似度约为 \(\cos\approx 0.22\)。这是因为决策存在于两个近乎正交的行的差异中,其中每个行的小旋转都会变成它们差异的大扰动。“权重看起来没问题”和“决策翻转了”都是真的。我们通过四种方式拒绝了量化损伤源于频率偏斜的稀有 token 分布的假设:
We evaluate the decision as a single logit comparison, \(\text{margin}=h\cdot(v_{\text{stop}}-v_{\text{cont}})\. As </think> is one token, everything shared by the two unembedding rows cancels out. After quantization, the </think> row provides a cosine similarity of 0.9961 to the original. The vector barely moves, however 2.45% of decisions flip due to the other vectors providing \(\cos\approx 0.22\). This is because the decision lives in the difference of two nearly-orthogonal rows, where a small rotation of each becomes a large perturbation of their difference. “The weights look fine” and “the decision flipped” are both true. We rejected a hypothesis of quantization damage from frequency-skewed rare-token distribution in four ways:
行最大绝对值的跨度为 2.84 binades(p99.9 处为 2.12 binades),而 FP8 约有 17.8 可用,因此在逐块尺度上没有可回收的空间。
Row-amax spread is 2.84 binades (2.12 binades at p99.9) against FP8's roughly 17.8 usable, so there is nothing to reclaim on per-block scale.
测试了从组大小为 1 到 64 个元素的误差,每种格式的汇总误差在小数点后四位完全相同,即 FP8 中为 0.0265,FP4 中为 0.0952。
Tested error ranging from group size of 1 through 64 elements, and the pooled error is identical to four decimal places per each format, i.e., 0.0265 in FP8, and 0.0952 in FP4.
按词汇频率四分位数划分的每行中位数跨度为 5.222 / 5.246 / 5.252 / 5.274 binades。换言之,在 131,072 个 token 上仅有 0.05 binade 的漂移。所有配置中均无死行,且相对量化误差最低范数十分位数并不比最高范数十分位数差:0.0941 对 0.0952。误差在整个词汇表上均匀分布。
Per-row median spread by vocabulary-frequency quartile is 5.222 / 5.246 / 5.252 / 5.274 binades. In other words, a 0.05-binade drift across 131,072 tokens. There are zero dead rows in every configuration, and the lowest-norm decile of relative quantization error is no worse than the highest: 0.0941 versus 0.0952. The error is uniformly distributed across the vocabulary.
NVFP4 在 66.9% 的块上越过其冲刷线,冲刷掉 6.8% 的元素,但这种冲刷在任何密度水平上都不携带词表结构。无论其代价如何,代价都遍布各处,因此 FP4 量化无法产生集中于稀有 token 的损失。
NVFP4 crosses its flush line on 66.9% of blocks, flushing 6.8% of elements, but this flush carries no vocabulary structure at any density level. Whatever it costs, it costs everywhere, so FP4 quantization cannot produce a loss concentrated in rare tokens.
剩余的假设是逐元素网格:E4M3 携带三个尾数位,提供 12.5% 步长的分辨率(\(1 \rightarrow 1.125)\),而 E2M1 只携带一个尾数位,提供 50% 步长的分辨率(\(1 \rightarrow 1.5)\)。为支持这一假设,我们模拟了双通道 FP4 权重,其中第二个通道校正残留在 \(\text{amax}/24\) 冲刷线以下的残差。这一校正完全弥补了差距:在采用线性 E2M1 残差时,KL 降至 \(0.00117\),而逐张量 FP8 为 \(0.00134\),且与 BF16 的 top-1 一致率达到 \(0.9951\)。降低 lm_head 中的量化误差仍有待探索,但这表明反嵌入层的数值敏感性局限于 FP8 精度之内。
The remaining hypothesis is per-element grid: E4M3 carries three mantissa bits providing a resolution of 12.5% step size (\(1 \rightarrow 1.125)\), whereas E2M1 carries one, providing a resolution of 50% step size (\(1 \rightarrow 1.5)\). To support this hypothesis, we emulated two-pass FP4 weights, where a second channel corrects the residual left below the \(\text{amax}/24\) flush line. This correction recovers the gap entirely: with a linear E2M1 residual the KL falls to \(0.00117\) against per-tensor FP8's \(0.00134\), and top-1 agreement with BF16 reaches \(0.9951\). Reducing the quantization error in lm_head is still open to exploration, but this suggests that the unembedding's numerical sensitivity is confined in FP8 precision.
同一套量化方案在 RL 后训练之前会加剧终止失败,而在其后则减轻了该失败。然而,它并不改变模型固有的结构性效应。在跨越 RL 前与 RL 后训练的三个代表检查点上,停止与继续的几何结构在 1% 以内保持不变(余弦 0.997,margin 0.996,噪声-裕度比 1.001),而终止行为却发生了显著变化。结合在路由专家中变化不超过 0.007 的贡献分布,可以给出一个清晰的结论:RL 后训练并未移动 lm_head 或专家贡献结构;它改变的是隐藏状态所指向的位置。因此,精度建议可以用结构性的语言来表述,但其 margin 保护必须依赖于具体检查点。本研究的局限性如下:
The same quantization recipe, which aggravated the termination failure before RL post-training, reduced it afterwards. However, it does not modify the model's inherent structural effect. Across three checkpoints spanning pre-RL and post-RL training, the stop-versus-continue geometry is invariant to within 1% (cosine 0.997, margin 0.996, noise-to-margin 1.001), while termination behavior changed significantly. Together with the contribution profile, which is invariant to within 0.007 in routed experts, this gives a clean statement: RL post-training did not move the lm_head or the expert contribution structure; it changed where the hidden state points. A precision recommendation can therefore be stated in structural terms, but its margin protection must be checkpoint-dependent. Here are limitations of this study:
关于终止失败的权重与激活归因,是从一个界推断而来的:我们并未对 W16A4 进行实验。
The weight-versus-activation attribution for the termination failure is an inference from a bound: we did not experiment with W16A4.
专家 GEMM 输出误差与基准测试的冗长度是两项独立的测量,在符号和量级上保持一致。没有任何证据将某一具体位移追溯到某一具体的额外 token;该关系是自洽的,而非已证明的因果。
Expert GEMM output error and benchmark verbosity are two separate measurements, consistent in sign and magnitude. Nothing traces a specific displacement to a specific extra token; the relationship is coherent, not demonstrated causal.
注意力分数路径的稀释是在干净权重上作为结构来测量的,而非作为扰动响应。
The attention score-path dilution is measured on clean weights as structure, not as a perturbation response.
本消融研究中的每一种配置,包括 BF16 参考配置,都服务于 FP8 KV cache;因此阶梯稀释在基线中同样存在,而投影增量是在已被稀释的分数之上测量的。
Every configuration in this ablation study, including the BF16 reference, serves an FP8 KV cache; the ladder dilution is therefore present in the baseline too, and the projection deltas are measured on top of an already-diluted score.
MRCR-128k 的消歧损失是相对于一个采用 FP8 KV cache 的未量化参考来测量的,但量化使用的是完整配置:FP8 KV cache、FP4 路由专家以及 FP4 Mamba 输出投影。在 Nemotron 3.5 Lightning 研究中,KV cache 被认定为损害载体,但 Nemotron 3.5 Super VL 消融并未涉及 BF16 KV 精度,因而无法对比其损害。
The disambiguation loss of MRCR-128k is measured against an unquantized reference with an FP8 KV cache, but the quantization uses a full configuration: FP8 KV cache, FP4 routed experts, and FP4 Mamba output projection. The KV cache is identified as the damage carrier in the Nemotron 3.5 Lightning study, but the Nemotron 3.5 Super VL ablation does not address BF16 KV precision to contrast its damage.
MRCR-128k 是该套件中唯一的消歧工具,因此结果的这一分支仅依赖单一基准。它每个任务重复三次,这足以测量解码噪声下限,但不足以支持超出所报告候选数量划分之外的逐层(per-stratum)论断。
MRCR-128k is the only disambiguation instrument in the suite, so that limb of the result rests on a single benchmark. It carries three repetitions per task, which is enough to measure the decoding-noise floor but not to support per-stratum claims beyond the reported candidate-count split.
反嵌入(unembedding)测量使用一个代理输入,即最后一层的 \(\text{norm}_{f}(\text{hidden}+\text{residual})\),也就是真实输入减去该层自身的 MoE 贡献。两项测试接收完全相同的输入,因此该比较精确地隔离了反嵌入量化;绝对比率并非模型在这些 token 上的真实输出。
The unembedding measurements use a proxy input, \(\text{norm}_{f}(\text{hidden}+\text{residual})\) at the last layer, i.e., the true input minus that layer's own MoE contribution. Both tests receive the identical input, so the comparison isolates unembedding quantization exactly; the absolute rates are not the model's true output on those tokens.
贡献分析使用两个固定的 token 窗口,每个训练阶段一个(pre-MOPD 与 MOPD)。在同一阶段内,BF16 与 FP4 的对比保持固定窗口以隔离权重,从而确立无论是否量化都不变的贡献分布。跨阶段比较同时改变了权重和窗口,因此它粗略地突显了一种结构性变化,但并未精确定位确切的根本原因。
The contribution analysis uses two fixed token windows, one per training stage (pre-MOPD versus MOPD). Within a stage, the BF16-versus-FP4 contrast holds the fixed window to isolate the weights, which establishes the invariant contribution profile with or without quantization. The cross-stage comparison changes both the weights and the window, so it loosely highlights a structural change but does not pinpoint the exact root cause.
我们对照以下基准测试对数据进行了检查:MATH-500、IFBench、KernelBench、MMLU-Pro、Terminal-Bench 2.0、Terminal-Bench 2.1、NV Chat Benchmark、Humanity's Last Exam、GPQA Diamond、CBRN、BBQ、TIR-Bench、SciCode、AA-Omniscience、AA-LCR、ICPC-Eval、PinchBench、ClawEval、SWE-bench Pro、SWE-bench Verified、MRCR、Tau-Bench、BrowseComp、CritPt、GDPval、IOI、MCP-Atlas、ProfBench、SWE-Atlas-QnA 以及 SWE-bench Multilingual。该套件既涵盖成熟的标准评测,也涵盖现代基准测试,其中 CoDeC 污染评测为选择提供了额外指导。
The data was checked against the following benchmarks: MATH-500, IFBench, KernelBench, MMLU-Pro, Terminal-Bench 2.0, Terminal-Bench 2.1, NV Chat Benchmark, Humanity's Last Exam, GPQA Diamond, CBRN, BBQ, TIR-Bench, SciCode, AA-Omniscience, AA-LCR, ICPC-Eval, PinchBench, ClawEval, SWE-bench Pro, SWE-bench Verified, MRCR, Tau-Bench, BrowseComp, CritPt, GDPval, IOI, MCP-Atlas, ProfBench, SWE-Atlas-QnA, and SWE-bench Multilingual. The suite spans both well-established standard evaluations and modern benchmarks, with CoDeC contamination evaluations providing additional guidance for the selection.
匹配结果集中在少数几个基准测试上。MATH-500 中有四分之一的问题被匹配到,IFBench 和 KernelBench 中约有 15% 的条目被匹配到,而 MMLU-Pro 在匹配问题的绝对数量上最多(1,254 个)。有八个基准测试没有任何匹配,它们大多是近期的智能体式与专家级基准测试。
The matches concentrate in a few benchmarks. A quarter of MATH-500 problems and about 15% of IFBench and KernelBench items were matched, and MMLU-Pro accounts for the largest absolute number of matched questions (1,254). Eight benchmarks, mostly recent agentic and expert-level ones, have no matches.
评判器接收一个固定的系统提示(如下),并将一个 JSON 对象 {“training_sample”:…,“benchmark_question”:…} 作为用户消息,后跟“仅返回 JSON 对象”。将两个文本作为 JSON 转义字符串传递,即使样本本身包含代码、标记或指令,也能保持边界明确。提示要求先给出推理再给出结论,并要求给出一个严重程度,该程度独立于标签,表示移除的重要性。表 41 总结了标签。
The judge receives a fixed system prompt (below) and, as the user message, a JSON object {"training_sample": …, "benchmark_question": …} followed by “Return only the JSON object.” Passing both texts as JSON-escaped strings keeps their boundaries unambiguous even when the sample itself contains code, markup, or instructions. The prompt asks for the reasoning before the verdict, and for a severity that expresses how important removal is, separately from the label. Table 41 summarizes the labels.
你是一名专家评估员,负责检测 LLM 训练数据中的基准污染。你会看到一个包含两个键的 JSON 对象:“training_sample”和 “benchmark_question”,每个键对应一个字符串。它们的内容本身可能包含代码、markdown 或 JSON——将每个字符串纯粹视为其自身的文本(JSON 字符串转义使边界明确)。仅根据这两个文本进行判断:决定将训练样本纳入训练数据是否会在基准问题上带来不公平的优势。
You are an expert evaluator detecting benchmark contamination in LLM training data. You are shown a single JSON object with two keys: "training_sample" and "benchmark_question", each a string. Their content may itself contain code, markdown, or JSON — treat each string purely as its own text (JSON string escaping makes the boundary unambiguous). Judge from the two texts alone: decide whether having the training sample in the training data would give an unfair advantage on the benchmark question.
选择唯一最合适的标签。它们大致按最强到最弱列出;如果多个适用,选择最能解释两者关系的那个:
Choose the single best-fitting label. They are listed roughly strongest-first; if several apply, pick the one that best explains the relationship between the two:
direct_copy:逐字或仅作微小修改地复现基准问题(和/或其答案)。
direct_copy: reproduces the benchmark question (and/or its answer) verbatim or with only minor edits.
paraphrase:重新表述与基准问题相同的具体问题,但更改了措辞或不相关的细节。
paraphrase: restates the same specific problem as the benchmark question, with wording or irrelevant details changed.
answer_leakage(答案泄漏):求解训练问题即可得到基准的答案——例如,基准是训练问题的特例、子问题或关键步骤。
answer_leakage: solving the training problem yields the benchmark's answer — e.g. the benchmark is a special case, sub-problem, or key step of the training problem.
method_leakage(方法泄漏):基准与训练问题用相同方法求解,仅在表面细节或数值上有所不同。
method_leakage: the benchmark is solved by the same method as the training problem, differing only in surface details or numeric values.
style_overlap(风格重叠):仅共享主题、领域或格式——属于一般知识,而非具体问题。
style_overlap: shares only the topic, domain, or format — general knowledge, not the specific problem.
no_contamination(无污染):不相关;任何相似性都是巧合。
no_contamination: unrelated; any similarity is coincidental.
然后以 0–4 的整数进行评分:
Then rate, as integers 0-4:
severity(严重程度):不公平优势的程度——为保持基准公平而移除该训练样本的重要性,从 0(保留无害)到 4(必须移除)。
severity: the degree of unfair advantage — how important it is to remove the training sample to keep the benchmark fair, from 0 (harmless to keep) to 4 (must be removed).
confidence(置信度):你对该严重程度评级的把握程度,从 0(不确定)到 4(确定)。这针对的是严重程度,而非你所选的标签。
confidence: how confident you are in that severity rating, from 0 (unsure) to 4 (certain). This concerns the severity, not which label you picked.
仅返回一个 JSON 对象,不要返回任何其他内容,并将推理放在最前面:{"reasoning": "<一句简洁的话>", "label": "<direct_copy|paraphrase|answer_leakage|method_leakage|style_overlap|no_contamination>", "severity": <0-4>, "confidence": <0-4>}
Return ONLY a JSON object and nothing else, with your reasoning first: {"reasoning": "<one concise sentence>", "label": "<direct_copy|paraphrase|answer_leakage|method_leakage|style_overlap|no_contamination>", "severity": <0-4>, "confidence": <0-4>}
真实数据不存在真实污染标签,因此我们在合成数据上评估该流程。我们取目标基准的所有问题,并对每个问题施加 13 种增强变换,每种变换有四个强度(0.25、0.5、0.75、0.9),用于缩放变化的剧烈程度。这些增强变换包括:
Ground-truth contamination labels do not exist for real data, so we evaluate the pipeline on synthetic data. We take all questions of a target benchmark and transform each with 13 augmentations, at four strengths (0.25, 0.5, 0.75, 0.9) that scale the intensity of the change. The augmentations are:
表层:逐字复制、字符噪声、大小写翻转、空白插入
surface: verbatim copy, character noise, case flipping, whitespace insertion
结构:裁剪到开头、中间或结尾,词丢弃,局部词重排,埋入其他问题之中
structural: cropping to the beginning, middle, or end, word dropout, local word reordering, burial among other questions
语义(由另一个 LLM 生成):复述(从轻微改写至完全重写风格)、翻译(翻译成越来越远的语言)以及抽象(对问题逐步泛化)。
semantic, generated by another LLM: paraphrase (from light rewording to a full restyle), translation (into increasingly distant languages), and abstraction (progressive generalization of the problem).
每个变换后的问题按构造即为正例。作为负例,我们加入来自同一领域不同基准的未修改问题,并选择难以与目标区分的问题:Terminal-Bench 2.1 使用 SWE-bench Pro(4,361 个正例,731 个负例),GPQA Diamond 使用 MMLU-Pro(9,702 个正例,12,032 个负例)。LLM 增强由与评判器不同的模型生成,因此评判器无法识别自己的改写。
Every transformed question is a positive by construction. As negatives we add unmodified questions from a different benchmark in the same domain, chosen to be hard to tell apart from the target: SWE-bench Pro for Terminal-Bench 2.1 (4,361 positives, 731 negatives) and MMLU-Pro for GPQA Diamond (9,702 positives, 12,032 negatives). The LLM augmentations are generated with a model different from the judge, so that the judge cannot recognize its own rewrites.
被移除的样本说明了为什么污染不能简化为字符串匹配。我们描述了在评审决策中发现的几种模式,而没有复现基准项目。
The removed samples illustrate why contamination cannot be reduced to string matching. We describe a few patterns found in the judge's decisions without reproducing the benchmark items.
翻译和重新表述的问题。几个 MATH-500 问题以中文翻译的形式出现在训练数据中,有些还被重新表述为要求从原始答案推导出的量(例如一个分数的分子和分母之和)。这样的样本与基准几乎没有共享的 n-gram,但仍然教会了答案。
Translated and reformulated problems. Several MATH-500 problems appear in the training data translated into Chinese, some also reformulated to ask for a quantity derived from the original answer (e.g. the sum of the numerator and denominator of a fraction). Such samples share almost no n-grams with the benchmark, yet still teach the answer.
聊天提示的跨语言变体。一些基准提示重新出现在其他语言(中文、日语、西班牙语、波兰语、俄语、德语、法语)的 RLHF 提示集合中,有时数字被更改,例如一个广泛传播的短距离旅行谜题用 50 米代替了 500 米。
Cross-lingual variants of chat prompts. Some benchmark prompts reappear in RLHF prompt collections in other languages (Chinese, Japanese, Spanish, Polish, Russian, German, French), sometimes with changed numbers, as in a widely shared short-distance travel puzzle posed with 50 instead of 500 meters.
智能体数据中的基准任务。Terminal-Bench 任务描述几乎逐字出现在智能体 RL 提示中,前面加上环境设置说明,并且文件路径不同。
Benchmark tasks inside agentic data. Terminal-Bench task descriptions appear nearly verbatim inside agentic RL prompts, prefixed with environment-setup instructions and with different file paths.
专家级问题。来自 Humanity's Last Exam 的问题以副本、释义或解决方案包含基准答案作为中间步骤的问题的形式出现在数学和工具集成推理数据中。
Expert-level questions. Questions from Humanity's Last Exam appear in math and tool-integrated reasoning data as copies, paraphrases, or problems whose solution contains the benchmark answer as an intermediate step.
上述所有案例均被阻止进入训练数据混合集。
All those cases were prevented from entering the training mix.
召回率受限于候选选择。评判器仅对由低成本信号选出的配对进行裁决。若某个受污染样本不在某问题的前若干候选之中,或低于相似度阈值,则该样本总会被保留。
Recall is bounded by candidate selection. The judge only rules on pairs selected by the cheap signals. A contaminated sample that is not among a question's top candidates, or that falls below the similarity gate, is always kept.
仅对问题陈述进行评分。源于使用同一组答案或其他产物的污染不会被检测到。
Only the problem statement is scored. Contamination that stems from using the same set of answers or other artifacts is not detected.
该评估只是一种代理。这些增强操作近似了实践中污染产生的各种方式,但并未穷尽它们。
The evaluation is a proxy. The augmentations approximate, but do not exhaust, the ways contamination arises in practice.