We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemotron 3 Super is the first model in the Nemotron 3 family to 1) be pre-trained in NVFP4, 2) leverage LatentMoE, a new Mixture-of-Experts architecture that optimizes for both accuracy per FLOP and accuracy per parameter, and 3) include MTP layers for inference acceleration through native speculative decoding. We pre-trained Nemotron 3 Super on 25 trillion tokens followed by post-training using supervised fine tuning (SFT) and reinforcement learning (RL). The final model supports up to 1M context length and achieves comparable accuracy on common benchmarks, while also achieving up to 2.2x and 7.5x higher inference throughput compared to GPT-OSS-120B and Qwen3.5-122B, respectively. Nemotron 3 Super datasets, along with the base, post-trained, and quantized checkpoints, are open-sourced on HuggingFace.
核心贡献 · Key contributions
首个在 NVFP4 低精度下预训练的混合 Mamba-Attention MoE 模型，展示了在 25T token 上的稳定训练。 First hybrid Mamba-Attention MoE model pre-trained in NVFP4 low precision, demonstrating stable training over 25T tokens.
引入 LatentMoE 架构，通过潜在空间专家路由优化每 FLOP 和每参数的准确率。 Introduces LatentMoE architecture optimizing accuracy per FLOP and per parameter via latent-space expert routing.
共享权重的多 Token 预测（MTP）实现原生推测解码，推理吞吐量提升高达 7.5 倍。 Multi-Token Prediction (MTP) with shared-weight heads enables native speculative decoding, improving inference throughput up to 7.5x.
两阶段 SFT 结合每对话归一化，在增强推理能力的同时保持长输入短输出性能。 Two-stage SFT with per-conversation normalization preserves long-input-short-output performance while enhancing reasoning.
跨 21 个多样化环境的大规模智能体强化学习，配备弹性基础设施以处理长周期工具使用任务。 Large-scale agentic RL across 21 diverse environments with resilient infrastructure for long-horizon tool-use tasks.
在 HuggingFace 上开源基础、后训练和量化检查点，以及预训练和后训练数据集。 Open-sourced base, post-trained, and quantized checkpoints along with pretraining and post-training datasets on HuggingFace.
局限 · Limitations
NVFP4 预训练因下溢导致零值权重梯度增加，但准确率未受影响。 NVFP4 pretraining increases zero-valued weight gradients due to underflow, though accuracy is not degraded.
检查点合并收益在长衰减调度下减弱；仅短周期合并仍有帮助。 Checkpoint merging gains diminish with long decay schedules; only short-horizon merging remains beneficial.
长上下文连续预训练可能轻微降低数学基准；通过交替 1M 和 4K 序列缓解。 Long-context CPT may slightly degrade math benchmarks; mitigated by alternating 1M and 4K sequences.