DeepSeek-V4.1-Flash：突破 KV 缓存压缩的极限

DeepSeek-V4.1-Flash：突破 KV 缓存压缩的极限 DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression

长时程智能体的广泛采用，使模型工作负载日益呈现输入密集型特征。尽管已有工作大幅降低了长上下文计算的开销，预填充（prefill）的计算成本依然高昂，而庞大的 KV 缓存持续对 HBM 与 SSD 容量以及数据传输带宽造成压力。这些算力、存储与带宽需求共同构成了进一步降低部署成本的主要瓶颈。为应对这一挑战，我们提出 DeepSeek-V4.1-Flash——一个多模态混合专家（MoE）模型，其骨干参数规模为 552B，支持最长一百万个 token 的上下文。凭借其因果编码器-解码器（CED）架构，该模型在解码阶段每个 token 激活 16B 参数，而在预填充阶段仅激活 8B 参数，显著提升了智能体式工作负载的成本效率。为突破 KV 缓存压缩的极限，DeepSeek-V4.1-Flash 将压缩稀疏注意力 2（CSA2）中的跨层 KV 缓存复用与 FP4 KV 缓存相结合。这些设计将其全局 KV 缓存占用（始终位于 HBM）降至每 token 890 字节，约为 DeepSeek-V4-Flash 相应占用的 1/4。此外，通过一项名为 SWA Bounded Replay 的专用部署优化，DeepSeek-V4.1-Flash 将其持久化 KV 缓存占用（始终位于 SSD 或主机内存）降至 DeepSeek-V4-Flash 的约 1/8。尽管 KV 缓存占用大幅缩小，该模型的性能仍显著优于基线。此外，我们精简了 DeepSeek-V4 架构，并引入了若干高效的架构扩展。我们在包含 45T token 的多模态语料上对 DeepSeek-V4.1-Flash 进行预训练，并开展全面的后训练，使其在多样化的纯文本与多模态智能体式场景中均取得强劲表现。模型检查点见 https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash。

The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy. Although prior work has substantially reduced the cost of long-context computation, prefill remains computationally expensive, and large KV caches continue to strain HBM and SSD capacity and data-transfer bandwidth. Together, these compute, storage, and bandwidth demands constitute the primary bottleneck to further lowering deployment costs. To address this challenge, we introduce DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens. With its Causal Encoder-Decoder (CED) architecture, the model activates 16B parameters per token during decode but only 8B parameters during prefill, substantially improving cost efficiency for agentic workloads. To push the limits of KV cache compression, DeepSeek-V4.1-Flash combines cross-layer KV cache reuse in Compressed Sparse Attention 2 (CSA2) with FP4 KV caching. These designs reduce its global KV cache footprint (always in HBM) to 890 bytes per token, roughly 1/4 of the corresponding footprint of DeepSeek-V4-Flash. Further, through a dedicated deployment optimization known as SWA Bounded Replay, DeepSeek-V4.1-Flash reduces its persistent KV cache footprint (always on SSD or in host memory) to roughly 1/8 of that of DeepSeek-V4-Flash. Despite its much smaller KV cache footprint, the model delivers substantially better performance than the baseline. In addition, we streamline the DeepSeek-V4 architecture and introduce several efficient architectural extensions. We pretrain DeepSeek-V4.1-Flash on a multimodal corpus comprising 45T tokens and conduct comprehensive post-training, yielding strong performance across diverse text-based and multimodal agentic scenarios. Model checkpoints are available at https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash.

1 引言 1 Introduction

近年来，长时程智能体的应用迅速扩展，使得超长上下文处理成为日益重要的模型工作负载。支撑此类工作负载不仅需要高效的长序列处理，还需要对大规模 KV 缓存进行持久化存储、复用与传输。因此，KV 缓存管理已成为模型部署的一项基础能力，同时在计算、存储与通信方面带来了巨大挑战。此前稀疏注意力机制方面的进展已显著降低了长序列处理的计算开销，使得持久化存储与数据搬运日益成为突出的瓶颈。

Applications of long-horizon agents have expanded rapidly in recent years, making ultra-long-context processing an increasingly important model workload. Supporting such workloads requires not only efficient long-sequence processing, but also the persistent storage, reuse, and transfer of large KV caches. KV cache management has therefore become a foundational capability for model deployment, while introducing substantial challenges across computation, storage, and communication. Prior advances in sparse attention have significantly reduced the computational cost of long-sequence processing, making persistent storage and data movement increasingly prominent bottlenecks.

具体而言，DeepSeek-V4 将覆盖完整上下文的全局注意力分支与局部滑动窗口注意力机制（SWA）相结合。全局分支维护全局 KV，包括主 KV 与索引器 K，而 SWA 维护局部 KV 状态。在窗口大小固定时，SWA KV 存储量与序列长度无关且有上界。对于足够长的序列，全局 KV 因此主导运行时 KV 占用，而后者受 HBM 容量约束。此外，部分 KV 会被持久化以用于前缀复用，称为持久化 KV 缓存，其受 SSD 与主机内存容量约束。I/O 与互连带宽也限制了缓存的迁移与加载。这些约束共同限制了服务吞吐量，增加了部署成本，并最终阻碍了智能体在更长任务时程与更广泛应用场景中的部署与采用。

Specifically, DeepSeek-V4 combines a global attention branch spanning the full context with local Sliding-Window Attention (SWA). The global branch maintains global KV, comprising main KV and indexer K, while SWA maintains local KV states. For a fixed window size, SWA KV storage is bounded independently of sequence length. For sufficiently long sequences, global KV therefore dominates the runtime KV footprint, which is constrained by HBM capacity. In addition, certain KV are persisted for prefix reuse, referred to as persistent KV caches, which are constrained by SSD and host memory capacity. I/O and interconnect bandwidth also limit cache migration and loading. Together, these constraints limit serving throughput, increase deployment costs, and ultimately hinder the deployment and adoption of agents over longer task horizons and across broader application scenarios.

因此，进一步降低 KV 缓存占用对于缓解存储与通信瓶颈、降低长上下文服务成本至关重要。为此，我们开发了 DeepSeek-V4.1-Flash，一个面向更激进 KV 缓存压缩而设计的多模态混合专家（MoE）模型。DeepSeek-V4.1-Flash 拥有 552B 骨干参数，原生支持多模态输入，并可容纳多达一百万个 token 的上下文。我们采用因果编码器-解码器（CED）架构，其中解码器全局 KV 由最终编码器隐藏状态投影而来。该设计使模型在预填充阶段每 token 激活 8B 参数，在解码阶段激活 16B 参数，对于输入密集的智能体式场景尤为经济高效。尽管规模远大于 DeepSeek-V4-Flash，DeepSeek-V4.1-Flash 在相同序列长度下仅需约 1/4 的运行时 KV 缓存存储与 1/8 的持久化 KV 缓存存储。此外，DeepSeek-V4.1-Flash 的整体性能优于 DeepSeek-V4-Flash。

Further reducing the KV cache footprint is therefore critical to alleviating storage and communication bottlenecks and lowering the cost of long-context serving. To this end, we develop DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model designed for more aggressive KV cache compression. DeepSeek-V4.1-Flash has 552B backbone parameters, natively supports multimodal inputs, and accommodates contexts of up to one million tokens. We adopt a Causal Encoder-Decoder (CED) architecture, in which decoder global KV is projected from the final encoder hidden states. This design enables the model to activate 8B parameters per token during prefill and 16B during decode, which is particularly cost-effective for input-heavy agentic scenarios. Despite being considerably larger than DeepSeek-V4-Flash, DeepSeek-V4.1-Flash requires only approximately 1/4 as much runtime KV cache storage and 1/8 as much persistent KV cache storage at the same sequence length. Moreover, DeepSeek-V4.1-Flash delivers better overall performance than DeepSeek-V4-Flash.

这一级别的 KV 缓存压缩是通过模型架构、缓存精度与部署策略的联合优化实现的。从概念上讲，DeepSeek-V4 可视为一个基于 SWA 的局部处理骨干，并辅以压缩的全局上下文。这一视角促使我们专注于简化全局分支，同时基本保留局部注意力设计。在架构层面，我们设计了压缩稀疏注意力机制 2（CSA2），对全局 KV（包括主 KV 与索引器 K）以及 Top-K 索引应用跨层复用，从而大幅降低 KV 缓存存储。CSA2 具有三种静态分配的模式：Full、Reindex 与 Reuse。Full 模式生成全局 KV 并执行索引。Reindex 模式复用前一层全局 KV，并使用自身的索引器 Q 对共享的索引器 K 重新打分并选取新的 Top-K 索引。Reuse 模式复用前一层的全局 KV 与 Top-K 索引，直接执行稀疏注意力。在三种模式中，每一层都保留自身的全局 Q 与 SWA KV。共享全局 KV 与索引器 K 减少了重复的缓存存储。此外，与采用压缩稀疏注意力机制（CSA）–重度压缩注意力机制（HCA）混合架构的 DeepSeek-V4 不同，DeepSeek-V4.1-Flash 使用纯 CSA2。在缓存精度层面，我们在训练期间使用 FP4 全局 KV 缓存，性能仅有轻微下降。CSA2 与 FP4 KV 缓存共同将全局 KV 缓存存储降至 DeepSeek-V4-Flash 的约 1/4，如图 1(b) 所示。在部署层面，DeepSeek-V4.1-Flash 与 DeepSeek-V4 一样，在每一层都使用滑动窗口注意力机制（SWA）。在 DeepSeek-V4 中，我们采用混合策略，在持久化 SWA KV 缓存的存储成本与精确重建所需的计算之间取得平衡。精确重建需要重放最近的 \(L \times n_{\mathrm{win}}\) 个 token，其中 \(L\) 为层数，\(n_{\mathrm{win}}\) 为 SWA 窗口大小。在 DeepSeek-V4.1-Flash 中，我们引入了 SWA 有界重放，仅通过重放最近的 \(n_{\mathrm{win}}\) 个 token 来近似重建所需的 SWA KV 状态。实验表明，这仅带来可忽略的性能下降。这一发现建立了新的存储–计算权衡，使我们能够避免将 SWA KV 缓存持久化到 SSD，同时仅产生少量预填充重计算。借助 SWA 有界重放，持久化 KV 缓存占用进一步降至 DeepSeek-V4-Flash 的约 1/8。这些优化共同极大缓解了 HBM 与 SSD 容量压力，降低了部署成本，并为更大规模的部署铺平了道路。

This level of KV cache compression is achieved through joint optimizations in model architecture, cache precision, and deployment strategy. Conceptually, DeepSeek-V4 can be viewed as an SWA-based local-processing backbone augmented with compressed global context. This perspective motivates us to focus on simplifying the global branch while largely preserving the local attention design. At the architectural level, we design Compressed Sparse Attention 2 (CSA2), which applies cross-layer reuse to global KV (including main KV and indexer K) and Top-K indices to substantially reduce KV cache storage. CSA2 has three statically assigned modes: Full, Reindex, and Reuse. Full Mode generates global KV and performs indexing. Reindex Mode reuses the global KV from a preceding layer, and uses its own indexer Q to rescore the shared indexer K and select fresh Top-K indices. Reuse Mode reuses both global KV and the Top-K indices in a preceding layer, and directly performs sparse attention. In all three modes, each layer retains its own global Q and SWA KV. Sharing global KV and indexer K reduces duplicated cache storage. In addition, different from DeepSeek-V4 that employs the Compressed Sparse Attention (CSA)–Heavily Compressed Attention (HCA) hybrid architecture, DeepSeek-V4.1-Flash uses pure CSA2. At the cache-precision level, we use FP4 global KV caches during training with only marginal performance degradation. Together, CSA2 and FP4 KV caching reduce global KV cache storage to approximately 1/4 of that of DeepSeek-V4-Flash, as shown in Figure 1(b). At the deployment level, DeepSeek-V4.1-Flash, like DeepSeek-V4, uses Sliding-Window Attention (SWA) in every layer. In DeepSeek-V4, we use a hybrid strategy to balance the storage cost of persisting SWA KV caches against the computation required for exact reconstruction. Exact reconstruction requires replaying the most recent \(L \times n_{\mathrm{win}}\) tokens, where \(L\) is the number of layers and \(n_{\mathrm{win}}\) is the SWA window size. In DeepSeek-V4.1-Flash, we introduce SWA Bounded Replay, which approximately reconstructs the required SWA KV states by replaying only the most recent \(n_{\mathrm{win}}\) tokens. Our experiments show that this incurs only negligible performance degradation. This finding establishes a new storage–computation trade-off, allowing us to avoid persisting SWA KV cache to SSD while incurring a small amount of prefill recomputation. With SWA Bounded Replay, the persistent KV cache footprint is further reduced to approximately 1/8 of that of DeepSeek-V4-Flash. Together, these optimizations greatly ease pressure on HBM and SSD capacity, reduce deployment costs, and pave the way for deployment at a larger scale.

作为 CED 与 CSA2 的补充，我们进一步精简了原始 DeepSeek-V4 架构。此外，我们将原始 mHC 设计升级为 Single-Pass mHC，并配套了 Mega-mHC 部署内核，相对于原始的四内核实现将激活内存流量减半。我们还集成了 Engram 条件记忆模块以增强模型能力。我们引入了 DSpark 投机解码架构，通过半自回归草稿生成与置信度调度验证来提升解码效率。综合所有架构改进，DeepSeek-V4.1-Flash 的单 token 解码 FLOPs 在不同上下文长度下几乎保持恒定。图 2 显示，将上下文长度扩展 256 倍（从 4K 到 1M）仅使其解码 FLOPs 增加 1/4，显著低于 DeepSeek-V4-Flash 所观察到的增长。

Complementing CED and CSA2, we further streamline the original DeepSeek-V4 architecture. Additionally, we upgrade the original mHC design to Single-Pass mHC, with an accompanying Mega-mHC deployment kernel that halves activation memory traffic relative to the original four-kernel implementation. Furthermore, we integrate the Engram conditional memory module to strengthen model capabilities. We also introduce the DSpark speculative decoding architecture to improve decoding efficiency through semi-autoregressive draft generation and confidence-scheduled verification. With all the architectural improvements combined, the single-token Decode FLOPs of DeepSeek-V4.1-Flash remain nearly constant across context lengths. Figure 2 shows that extending the context length 256-fold, from 4K to 1M, increases its Decode FLOPs by only 1/4, significantly less than the growth observed for DeepSeek-V4-Flash.

为了充分实现这些架构设计带来的 KV cache 压缩收益，并进一步提升训练与推理效率，我们针对 DeepSeek-V4.1-Flash 系统性地协同优化了训练基础设施与推理系统，确保高效且可扩展的大规模多模态训练与长上下文部署。训练基础设施支持视觉编码器的解耦执行、面向长序列的均衡图像分片，以及用于注意力机制复用的跨阶段共享状态管理。推理系统实现了 Encoder 与 Decoder 的 SWA Bounded Replay 路径。进一步的优化包括通信–计算重叠、分片的 Engram 嵌入表以及推理 kernel 融合。特别地，每个 CSA2 Reuse Mode 层在 prefill 阶段仅执行 15 个 kernel，在 decode 阶段仅执行 11 个。我们还在主机内存中将长生命周期的全局 KV 存储与短生命周期的 encoder SWA KV 分离，利用 bounded replay 近似重建缺失的 encoder SWA 状态。

To fully realize the KV cache compression benefits of these architectural designs and further improve training and inference efficiency, we systematically co-optimize the training infrastructure and inference system for DeepSeek-V4.1-Flash, ensuring efficient and scalable large-scale multimodal training and long-context deployment. The training infrastructure supports disaggregated vision-encoder execution, balanced image sharding for long sequences, and cross-stage shared-state management for attention reuse. The inference system implements Encoder and Decoder SWA Bounded Replay paths. Further optimizations include communication–computation overlap, sharded Engram embedding tables, and inference kernel fusion. In particular, each CSA2 Reuse Mode layer executes with only 15 kernels during prefill and 11 during decode. We also separate long-lived global KV storage from short-lived encoder SWA KV in host memory, using bounded replay to approximately reconstruct missing encoder SWA states.

在预训练阶段，我们在包含 45T tokens 的大规模多模态语料上训练 DeepSeek-V4.1-Flash。稀疏注意力机制在序列长度 64K 下从零开始训练，没有任何稠密注意力预热阶段。预训练后，模型具备原生多模态能力，并支持高达一百万 tokens 的上下文窗口。在我们的评估中，DeepSeek-V4.1-Flash-Base 在世界知识、推理和编码能力上与 DeepSeek-V4-Pro-Base 相当，并在留出评估上取得 5%–10% 的提升，而仅使用 1/3 的总参数和 1/4 的激活参数。这些结果共同凸显了其强大的参数效率，并反映了面向真实部署的训练数据质量提升。

During pre-training, we train DeepSeek-V4.1-Flash on a large-scale multimodal corpus comprising 45T tokens. Sparse attention is trained from scratch at a sequence length of 64K, without any dense attention warmup stages. After pre-training, the model possesses native multimodal capabilities and supports contexts of up to one million tokens. In our evaluations, DeepSeek-V4.1-Flash-Base achieves world knowledge, reasoning and coding abilities comparable to DeepSeek-V4-Pro-Base, and delivers 5%–10% improvements on held-out evaluations, using only 1/3 total parameters and 1/4 activated parameters. Together, these results highlight its strong parameter efficiency and reflect improvements in training data quality for real-world deployment.

在此基础模型之上，我们进行后训练以激发其推理与智能体式能力。与上述架构创新不同，我们的后训练没有引入任何算法创新：训练配方遵循监督微调（SFT）后接强化学习（RL）与同策略蒸馏（OPD）的标准范式，除 DeepSeek-V4 开发中已确立的实践外没有任何修改。所有实质性变化都在数据流水线中。我们开发了用于数据合成与环境构建的大规模自动化流水线，并逐步扩展强化学习中所使用的数据、任务和 rollout，从而将模型能力扩展到文本、多模态和智能体式领域。图 1(a) 总结了 DeepSeek-V4.1-Flash 在核心智能体式基准上的表现。我们的评估表明，尽管规模紧凑，DeepSeek-V4.1-Flash 展现出独特的能力画像：

Building on this base model, we conduct post-training to elicit its reasoning and agentic capabilities. In contrast to the architectural innovations described above, our post-training introduces no algorithmic innovation: the recipe follows the standard paradigm of supervised fine-tuning (SFT) followed by reinforcement learning (RL) and on-policy distillation (OPD), without any modification beyond well-established practice used in DeepSeek-V4 development. All substantive changes lie instead in the data pipeline. We develop large-scale automated pipelines for data synthesis and environment construction, and progressively scale the data, tasks, and rollouts employed during RL, thereby extending the model’s capabilities across textual, multimodal, and agentic domains. Figure 1(a) summarizes DeepSeek-V4.1-Flash’s performance on core agentic benchmarks. Our evaluation shows that, despite its compact size, DeepSeek-V4.1-Flash exhibits a distinctive capability profile:

**推理。** 该模型具备强大的推理能力，在数学和竞技编程等推理密集型基准上保持高准确率，与顶级开源模型（如 Kimi-K3 和 DeepSeek-V4-Pro）表现相当。

Reasoning. The model delivers strong reasoning ability, sustaining high accuracy on reasoning-intensive benchmarks such as mathematics and competitive programming, showing comparable performance with top open-source models, such as Kimi-K3 and DeepSeek-V4-Pro.

**智能体。** DeepSeek-V4.1-Flash 在 Terminal-Bench 2.1、DeepSWE v1.1 和 AutomationBench 等标准智能体式基准上达到与闭源前沿模型相当的性能。它已被证明完全能够处理日常编码任务和白领工作流。然而，在需要专家级领域知识的科学导向智能体式任务（如 Terminal-Bench 4.0）上，与巨型模型之间仍存在差距。

Agent. DeepSeek-V4.1-Flash achieves performance on par with closed-source frontier models across standard agentic benchmarks like Terminal-Bench 2.1, DeepSWE v1.1, and AutomationBench. It has proven fully capable of handling everyday coding tasks and white-collar workflows. However, a gap with giant models remains on science-oriented agentic tasks, such as Terminal-Bench 4.0, that require expert-level domain knowledge.

**多模态。** 在多模态领域，该模型在评估视觉推理和专业图表解读的基准上，超越了 Kimi-K3 等顶级开源竞争者。除正式指标外，它在真实世界的视觉智能体式工作流中也展现出实用价值，例如前端开发和办公自动化，在这些场景中它能够利用渲染的屏幕截图进行视觉检查和自我纠正。尽管如此，我们承认与巨型闭源系统相比，仍存在明显的整体性能差距。

**Multimodal.** Within the multimodal domain, the model surpasses top-tier open-source competitors like Kimi-K3 specifically on benchmarks evaluating visual reasoning and the interpretation of professional charts. Beyond formal metrics, it also exhibits practical utility in real-world visual agentic workflows, such as frontend development and office automation, where it can utilize rendered screen captures for visual inspection and self-correction. Nevertheless, we acknowledge that a distinct overall performance gap remains when compared to giant closed-source systems.

这些结果表明，DeepSeek-V4.1-Flash 已能在绝大多数基准上匹配闭源前沿模型，并能够完成超过 95% 的真实世界任务。同时，其较小的激活占用带来了低推理延迟和服务成本。因此，我们认为 DeepSeek-V4.1-Flash 在能力与效率之间提供了良好的权衡，可作为快速、经济的助手，支持广大用户的日常工作。总之，DeepSeek-V4.1-Flash 同时提升了模型智能和推理效率，并降低了部署成本。它大幅降低了大规模部署长时程智能体的成本门槛，并为其在更广泛场景中的采用创造了新机会。DeepSeek-V4.1-Flash 也是我们持续 Scaling（规模扩张）努力的新起点。在此基础上，我们将追求模型架构、预训练和后训练的联合 Scaling（规模扩张），以进一步探索模型智能的前沿。

These results indicate that DeepSeek-V4.1-Flash can already match closed-source frontier models on the vast majority of benchmarks, and is capable of completing over 95% of real-world tasks. Meanwhile, its small activation footprint yields low inference latency and serving cost. We therefore believe that DeepSeek-V4.1-Flash offers a favorable trade-off between capability and efficiency, and can serve as a fast, affordable assistant supporting the daily work of a broad population of users. In summary, DeepSeek-V4.1-Flash simultaneously improves model intelligence and inference efficiency while reducing deployment costs. It substantially lowers the cost barrier to deploying long-horizon agents at scale and creates new opportunities for their adoption across a broader range of scenarios. DeepSeek-V4.1-Flash also serves as a new starting point for our continued scaling efforts. Building on this foundation, we will pursue the joint scaling of model architecture, pre-training, and post-training to further explore the frontier of model intelligence.

2.1 概述 2.1 Overview

DeepSeek-V4.1-Flash 是一个多模态混合专家（MoE）Transformer，以图像和文本为输入，自回归地生成文本。其语言主干由 40 个因果 Transformer 层组成，组织为一个 20 层的因果编码器后接一个 20 层的解码器。除前两层仅使用滑动窗口注意力（SWA）外，每一层都同时包含全局注意力机制和滑动窗口注意力机制。一个视觉编码器和一个 MLP 投影器将图像转换为视觉嵌入，这些嵌入与文本嵌入一起被处理，多模态数据从语言模型预训练开始就被纳入。总体而言，DeepSeek-V4.1-Flash 拥有 552B 主干参数和 196B Engram 参数，在预填充阶段每个词元激活 8B 参数，在解码阶段激活 16B 参数。图 3 展示了 DeepSeek-V4.1-Flash 的整体架构。

DeepSeek-V4.1-Flash is a multimodal mixture-of-experts (MoE) Transformer that takes images and text as input and generates text autoregressively. Its language backbone comprises 40 causal Transformer layers, organized into a 20-layer causal encoder followed by a 20-layer decoder. Each layer incorporates both global attention and sliding window attention (SWA), except for the first two layers, which use SWA only. A vision encoder and an MLP projector convert images into visual embeddings that are processed jointly with text embeddings, with multimodal data incorporated from the start of language-model pre-training. Overall, DeepSeek-V4.1-Flash has 552B backbone parameters and 196B Engram parameters, activating 8B parameters per token during prefill and 16B during decode. Figure 3 illustrates the overall architecture of DeepSeek-V4.1-Flash.

因果编码器-解码器（CED）架构和压缩稀疏注意力机制 2（CSA2）解决了长上下文推理中互补的开销问题。CED 从编码器输出构建解码器的全局键值（KV）缓存，使大多数提示词元绕过完整的解码器计算，同时保留层局部的滑动窗口注意力机制。这几乎将预填充计算减半，降低了在上下文不断增长的智能体式工作负载中处理新输入或未缓存输入的成本。CSA2 在层间共享全局 KV 以减少缓存存储，并复用稀疏选择以减少索引工作。在解码器中，分层稀疏索引器将后续索引器限制在由较早索引器选出的候选池内，进一步减少每个查询评分的条目数量。

The Causal Encoder–Decoder (CED) architecture and Compressed Sparse Attention 2 (CSA2) address complementary costs of long-context inference. CED constructs the decoder’s global key-value (KV) cache from encoder outputs, allowing most prompt tokens to bypass full decoder computation while retaining layer-local sliding-window attention. This nearly halves prefill computation, lowering the cost of processing new or uncached inputs in agentic workloads with growing contexts. CSA2 shares global KV across layers to reduce cache storage and reuses sparse selections to reduce indexing work. In the decoder, a Hierarchical Sparse Indexer restricts later indexers to a candidate pool selected by an earlier indexer, further reducing the number of entries scored per query.

我们保留了 DeepSeekMoE 的共享专家和细粒度路由专家，并针对图像和文本词元引入了模态特定的负载均衡。单遍 mHC 修改了残差流混合，以实现更高效的内核融合，而 Engram 增加了稀疏访问的条件记忆。我们在主干预训练期间省略了 MTP 模块，并使用 DSpark 进行推测解码。我们在主干预训练阶段之后单独训练 DSpark。此外，我们将主 KV 缓存压缩为 FP4，以进一步减少存储开销。以下各节将描述这些组件及相应的优化变更。

We retain the shared and fine-grained routed experts of DeepSeekMoE, and introduce modality-specific load balancing for image and text tokens. Single-Pass mHC revises residual-stream mixing to enable more efficient kernel fusion, and Engram adds sparsely accessed conditional memory. We omit the MTP module during backbone pre-training and use DSpark for speculative decoding. We train DSpark separately after the backbone pre-training stage. Additionally, we compress the main KV cache to FP4 to further reduce storage overhead. The following sections describe these components and the corresponding optimization changes.

2.1.1 多模态架构 2.1.1 Multimodal Architecture

多模态输入通路包含一个视觉编码器和一个 MLP 投影器。对于每张输入图像，视觉编码器生成一个空间网格的视觉特征。随后，一个 \(3 \times 3\) 像素反洗牌操作沿通道维度重排每个局部邻域，在 MLP 投影器将特征映射到语言主干的隐藏维度之前降低空间分辨率。最后，生成的视觉嵌入被插入到输入嵌入序列中对应的图像 token 位置，并由语言主干与文本嵌入联合处理。

The multimodal input pathway comprises a vision encoder and an MLP projector. For each input image, the vision encoder produces a spatial grid of visual features. A \(3 \times 3\) pixel-unshuffle operation then rearranges each local neighborhood along the channel dimension, reducing the spatial resolution before the MLP projector maps the features to the hidden dimension of the language backbone. Finally, the resulting visual embeddings are inserted at the corresponding image-token positions in the input embedding sequence and processed jointly with text embeddings by the language backbone.

我们从头训练了一个名为 DeepSeek-ViT 的视觉编码器，以原生处理不同分辨率的图像。我们在 Vision Transformer 架构上构建 DeepSeek-ViT，并进行了若干修改。为适应任意分辨率的输入，我们用 2D-RoPE 替换了标准的绝对位置嵌入。为使 ViT 更贴近 LLM 的设计原则，我们将 patch 嵌入层的卷积替换为线性投影，以确保与 Muon 优化器兼容。我们还采用 RMSNorm 进行归一化，并使用 SwiGLU 作为激活函数。在将视觉特征输入 LLM 之前，我们应用 \(3 \times 3\) 下采样的像素反洗牌操作，将视觉 token 数量减少为原来的九分之一，从而有效支持高达约 \(1344 \times 1344\) 像素的输入分辨率。

We train a vision encoder named DeepSeek-ViT from scratch to natively process images at varying resolutions. We build DeepSeek-ViT on the Vision Transformer architecture with several modifications. To accommodate inputs of arbitrary resolutions, we replace standard absolute positional embeddings with 2D-RoPE. To align the ViT more closely with LLM design principles, we replace the patch embedding layer's convolution with a linear projection to ensure compatibility with the Muon optimizer. We also adopt RMSNorm for normalization and SwiGLU as the activation function. Before feeding visual features into the LLM, we apply a pixel-unshuffle operation with \(3 \times 3\) downsampling to reduce the visual token count by a factor of nine, effectively supporting input resolutions up to approximately \(1344 \times 1344\) pixels.

图像和文本 token 表现出不同的表示分布，并可能在 MoE 中引发不同的专家路由偏好。因此，平衡它们的总负载可能会掩盖模态特定的不平衡。为解决这一问题，我们扩展了无辅助损失负载均衡，为文本和图像 token 分别维护专家级的校正偏置。在路由过程中，每个 token 使用与其模态相关的校正偏置进行专家选择，同时保留原始路由分数以对所选专家的输出进行加权。在每个训练步骤之后，两组偏置根据各自的专家负载独立更新。这种设计平衡了每个模态内的专家利用率，并有助于稳定高效的多模态训练。

Image and text tokens exhibit distinct representation distributions and may induce different expert-routing preferences in MoEs. Balancing their aggregate load may therefore obscure modality-specific imbalance. To address this issue, we extend auxiliary-loss-free load balancing by maintaining separate expert-wise correction biases for text and image tokens. During routing, each token uses the correction biases associated with its modality for expert selection, while the original routing scores are retained for weighting the selected expert outputs. After each training step, the two sets of biases are updated independently according to their respective expert loads. This design balances expert utilization within each modality and contributes to stable and efficient multimodal training.

2.2 因果编码器-解码器（CED） 2.2 Causal Encoder-Decoder (CED)

在智能体式工作流中，频繁的工具调用会产生大量的预填充请求，当 KV 缓存未命中时会带来严重的计算开销。为缓解这一预填充瓶颈，我们受 YOCO 启发，提出了因果编码器-解码器（CED）架构。YOCO 通过让上半部分层直接共享下半部分层生成的 KV 缓存来减少预填充计算。在此概念基础上，CED 引入了一系列结构改进，以同时提升整体 KV 缓存容量和 KV 生成的计算深度。因此，CED 成功减少了近一半的预填充计算，同时保持了与基线相当的性能。

In agentic workflows, frequent tool calls generate extensive prefill requests, imposing severe computational overhead when KV caches miss. To alleviate this prefill bottleneck, we propose the Causal Encoder-Decoder (CED) architecture, inspired by YOCO. YOCO reduces prefill computation by allowing the upper half of the layers to directly share the KV cache generated by the lower half. Building upon this concept, CED introduces a series of structural improvements to enhance both the overall KV cache capacity and the computational depth of KV generation. Consequently, CED successfully reduces nearly half of the prefill computation while maintaining performance comparable to the baseline.

对于全局注意力机制，CED 将 Transformer 的底部 \(L/2\) 层视为因果编码器。对于上半部分层（即解码器，\(l>L/2\)），其 KV 条目并非来自各自的隐藏状态 \(H_{l}\)，而是使用层相关的投影权重（\(W_{l}^{KV}\) 和 \(W_{l}^{Z}\)）直接从第 \((L/2)\) 层的隐藏状态 \(H_{L/2}\) 投影得到：

For global attention, CED treats the bottom \(L/2\) layers of the Transformer as the causal encoder. For the upper half layers (i.e., the decoder, \(l>L/2\)), the KV entries are not derived from their respective hidden states \(H_{l}\). Instead, they are projected directly from the hidden state of the \((L/2)\)-th layer, \(H_{L/2}\), using layer-dependent projection weights (\(W_{l}^{KV}\) and \(W_{l}^{Z}\)):

其中 \(C\) 和 \(Z\) 分别表示 KV 条目及其对应的压缩权重。这一设计使 CED 在预填充阶段仅需计算前半部分层，从而以极小的计算成本获得上层的全局 KV 缓存。

where \(C\) and \(Z\) represent the KV entries and their corresponding compression weights, respectively. This design allows CED to compute only the first half of the layers during the prefill phase, acquiring the upper-layer global KV cache with minimal computational cost.

对于滑动窗口注意力机制（SWA），CED 在所有层中保持传统的逐层计算。具体而言，对于任意层 \(l\)，局部键和值直接来自当前层的隐藏状态 \(H_{l}\)。这一设计有效提升了局部 KV 生成的计算深度。然而，保持这种逐层计算需要 SWA 重放过程。在预填充阶段，为解码器计算 SWA KV 缓存需要额外处理 \(n_{\mathrm{win}}\times L/2\) 个 token（其中 \(n_{\mathrm{win}}\) 表示窗口大小）。对于每轮提示较短的交互，解码器中的这一计算开销变得不可忽略。幸运的是，先前工作表明 SWA 的实际有效感受野远小于理论上的 \(n_{\mathrm{win}}\times L/2\)。受此观察启发，我们引入了解码器 SWA 有界重放，仅对提示的最后 \(n_{\mathrm{win}}\) 个 token 进行预填充以用于 SWA 计算，从而显著降低计算成本。更多细节见第 3.2.2 节。

For sliding window attention (SWA), CED maintains the conventional layer-wise computation across all layers. Specifically, for any layer \(l\), the local keys and values are derived directly from the current layer’s hidden state \(H_{l}\). This design effectively increases the computational depth of local KV generation. However, maintaining this layer-wise computation necessitates an SWA replay process. During the prefill phase, computing the SWA KV cache for the decoder requires processing an additional \(n_{\mathrm{win}}\times L/2\) tokens (where \(n_{\mathrm{win}}\) denotes the window size). For multi-turn interactions with short prompts per turn, this computational overhead in the decoder becomes non-negligible. Fortunately, prior work has shown that the actual effective receptive field of SWA is much smaller than the theoretical \(n_{\mathrm{win}}\times L/2\). Motivated by this observation, we introduce Decoder SWA Bounded Replay, which only prefills the last \(n_{\mathrm{win}}\) tokens of the prompt for the SWA computation, thereby significantly reducing the computational cost. Further details are provided in Section 3.2.2.

总体而言，对于序列长度 \(N\gg n_{\mathrm{win}}\)，CED 将预填充复杂度从 \(\mathcal{O}(NL)\) 降低到 \(\mathcal{O}(NL/2+n_{\mathrm{win}}\times L/2)\approx\mathcal{O}(NL/2)\)，有效将整体计算量减半。

Overall, for a sequence length \(N\gg n_{\mathrm{win}}\), CED reduces the prefill complexity from \(\mathcal{O}(NL)\) to \(\mathcal{O}(NL/2+n_{\mathrm{win}}\times L/2)\approx\mathcal{O}(NL/2)\), effectively halving the overall computation.

2.3 压缩稀疏注意力机制 2（CSA2） 2.3 Compressed Sparse Attention 2 (CSA2)

服务长上下文需要同时控制 KV 缓存存储和注意力机制计算。这些成本可以从三个乘性维度上降低：条目大小，其中 GQA 减少了 KV 头的数量，MLA 在头之间共享一个小潜在表示；序列维度，其中每 \(m\) 个 token 被压缩成一个条目，如 DeepSeek-V4 中的 CSA 和 HCA；以及层维度，其中一些层重用其他层的缓存和选择，而不是保留自己的，或者完全被更高效的层替换。先前的工作表明，沿层维度的压缩是有效的：IndexCache 跨层重用 Top-K 索引以减少索引器计算；YOIO 计算一次稀疏路由并在所有层之间共享；HySparse 允许稀疏层重用密集层的 KV 缓存。然而，仅索引重用不能节省主 KV 存储，网络范围的路由共享限制了性能，混合设计仍然保留完整的注意力机制层；更重要的是，这些方法都没有覆盖所有三个乘性维度。

Serving long contexts requires controlling both KV cache storage and attention computation. These costs can be reduced along three multiplicative dimensions: the entry size, where GQA reduces the number of KV heads and MLA shares a small latent across heads; the sequence dimension, where every \(m\) tokens are compressed into one entry, like CSA and HCA in DeepSeek-V4; and the layer dimension, where some layers reuse the caches and selections of other layers instead of keeping their own, or are replaced altogether by more efficient layers. Prior work has shown that compression along the layer dimension is effective: IndexCache reuses Top-K indices across layers to cut indexer computation; YOIO computes the sparse routing once and shares it across all layers; and HySparse lets sparse layers reuse the KV cache of dense layers. However, index reuse alone saves no main KV storage, network-wide routing sharing limits performance, and hybrid designs still retain full attention layers; more importantly, none of these methods covers all three multiplicative dimensions.

CSA2 联合利用这三个维度：它跨层共享主 KV 和索引器 K，并允许层重用 Top-K 索引，缓存共享和索引重用解耦。它将这些重用策略与简化的压缩器以及分层稀疏索引器相结合，后者缩小了 Decoder 中后续索引层的搜索域。

CSA2 exploits the three dimensions jointly: it shares main KV and indexer K across layers and allows layers to reuse Top-K indices, with cache sharing and index reuse decoupled. It combines these reuse strategies with a simplified compressor and a Hierarchical Sparse Indexer that narrows the search domain of subsequent indexing layers in the Decoder.

与 CSA 类似，CSA2 包含一个轻量级索引器，使用索引器 Q 和索引器 K 对主 KV 条目进行评分，并为每个查询选择 Top-K 条目。每个 Q 关注所选条目以及层局部滑动窗口 KV（SWA KV）。CSA2 还包括未压缩主 KV 设置作为压缩比为 1 的特例。同时，CSA2 简化了压缩器和索引器。在 CSA 中，压缩比 \(m\) 从 \(2m\) 个原始 KV 缓存条目生成每个主 KV 条目，相邻压缩条目的源条目重叠。它还包括绝对位置嵌入，以在压缩期间编码这些 \(2m\) 个条目的位置。CSA2 移除了这种重叠和绝对位置嵌入。此外，CSA2 通过投影主 KV 条目获得索引器 K，取代了 CSA 从隐藏状态单独压缩的路径。这两种设计都简化了实现并提高了训练效率。

Similar to CSA, CSA2 includes a lightweight indexer that scores the main KV entries using indexer Q and indexer K and selects the Top-K entries for each query. Each Q attends to the selected entries together with the layer-local sliding-window KV (SWA KV). CSA2 also includes the uncompressed main KV setting as a special case with a compression ratio of 1. Meanwhile, CSA2 simplifies both the compressor and the indexer. In CSA, a compression ratio of \(m\) produces each main KV entry from \(2m\) original KV cache entries, with overlapping source entries for adjacent compressed entries. It also includes absolute positional embedding to encode the positions of these \(2m\) entries during compression. CSA2 removes this overlap and absolute positional embedding. In addition, CSA2 obtains indexer K by projecting main KV entries, replacing CSA’s separate compression path from hidden states. Both designs simplify the implementation and increase the training efficiency.

2.3.1 节和 2.3.2 节分别描述了跨层重用策略和分层稀疏索引器。

Sections 2.3.1 and 2.3.2 describe the cross-layer reuse strategies and the Hierarchical Sparse Indexer, respectively.

2.3.1 跨层 KV 与索引复用 2.3.1 Cross-Layer KV and Index Reuse

每个 CSA2 层被静态分配为三种模式之一：Full、Reindex 或 Reuse。在三种模式下，该层都计算自己的 query 和 SWA KV，并将它们与选定的 main KV 条目一起使用，以产生新的注意力输出。这些模式在获取 main KV、indexer K 和 Top-K 索引的方式上有所不同。图 4 展示了这三种模式。

Each CSA2 layer is statically assigned one of three modes: Full, Reindex, or Reuse. In all three modes, the layer computes its own query and SWA KV and uses them together with the selected main KV entries to produce a new attention output. The modes differ in how they obtain main KV, indexer K, and Top-K indices. Figure 4 illustrates the three modes.

Full 模式。该层计算自己的 main KV 和 indexer Q，从该 main KV 投影出 indexer K，并运行 indexer 以产生新的 Top-K 索引。因此，它执行完整的 CSA2 计算路径，并具有与 DeepSeek-V4 中完整 CSA 层相同的组件职责。

Full Mode. The layer computes its own main KV and indexer Q, projects indexer K from that main KV, and runs the indexer to produce fresh Top-K indices. It therefore executes the complete CSA2 computation path and has the same component responsibilities as a complete CSA layer in DeepSeek-V4.

Reindex 模式。该层复用来自前一层的最新可用 main KV 及其对应的 indexer K。indexer 计算自己的 query，对复用的键重新评分，并产生新的 Top-K 索引。这允许稀疏选择在不同层之间变化，同时 main KV 和 indexer K 保持共享。

Reindex Mode. The layer reuses the most recent available main KV from a preceding layer together with its corresponding indexer K. The indexer computes its own query, rescores the reused keys, and produces fresh Top-K indices. This allows the sparse selection to change across layers while main KV and indexer K remain shared.

Reuse 模式。该层复用最新可用的 main KV 以及前一层在 Full 或 Reindex 模式下针对该 main KV 计算的最新 Top-K 索引。它使用此选择执行注意力，而不计算 indexer Q 或评估索引分数。

Reuse Mode. The layer reuses the most recent available main KV and the latest Top-K indices computed against that main KV by a preceding layer in Full or Reindex Mode. It performs attention using this selection without computing indexer Q or evaluating index scores.

共享 main KV 和 indexer K 可减少缓存存储，而复用 Top-K 索引可避免额外的 indexer 计算。Reindex 模式在保持缓存共享的同时，允许所选条目在不同层之间变化。当 CSA2 与 CED 结合时，分配给 Full 模式的解码器层从第 \((L/2))\ 层的隐藏状态（即因果编码器的最后一层）计算自己的全局 KV。Reindex 和 Reuse 模式保持不变。

Sharing main KV and indexer K reduces cache storage, while reusing Top-K indices avoids additional indexer computation. Reindex Mode preserves cache sharing while allowing the selected entries to change across layers. When CSA2 is combined with CED, the decoder layer assigned to Full Mode computes its own global KV from the hidden state of the \((L/2))\-th layer, i.e. the last layer of the causal encoder. The Reindex and Reuse Modes are unchanged.

2.3.2 分层稀疏索引器 2.3.2 Hierarchical Sparse Indexer

跨层索引复用减少了索引器评估的次数，但剩余的索引器仍会对整个因果可见上下文进行评分。对于极长的上下文，这一开销仍是主要的计算瓶颈。先前的工作通过在词元级索引之前对池化块表示进行评分和剪枝，引入了索引器稀疏性。我们发现，在解码器中，来自较浅层索引器的信息可以自然地用于限制较深层索引器所考虑的候选，而无需添加任何额外状态。因此，我们引入了分层稀疏索引器，仅用于 CED 的解码器中，以减少解码过程中的重复评分。对于每个查询，第一个分配为 Full Mode 的层构建一个候选池，后续重新索引层将其用作搜索域。对于固定的候选池大小，这将较深层索引器的每查询开销从与上下文长度成线性关系变为常数。该机制是训练感知的，并在后训练中引入：候选限制在训练和推理期间以相同方式应用，因此较深层索引器在其推理时使用的相同搜索域下进行优化。图 5 展示了这一过程。

Cross-layer index reuse reduces the number of indexer evaluations, but the remaining indexers still score the full causally visible context. For extremely long contexts, this cost remains a major computational bottleneck. Prior work introduced indexer sparsity by scoring and pruning pooled block representations before token-level indexing. We find that in the decoder, information from shallower indexers can naturally be used to restrict the candidates considered by deeper indexers without adding any extra state. We therefore introduce the Hierarchical Sparse Indexer, which is used only in the decoder of CED to reduce this repeated scoring during decode. For each query, the first layer assigned to Full Mode constructs a candidate pool that later re-indexing layers use as their search domain. For a fixed candidate-pool size, this changes the per-query cost of deeper indexers from linear in context length to constant. The mechanism is training-aware and introduced in post-training: the candidate restriction is applied identically during training and inference, so deeper indexers are optimized under the same search domain they use at inference. Figure 5 illustrates this process.

第一个 Full Mode 层对所有因果可见的主 KV 位置进行评分，并为其自身的注意力机制生成 Top-K 索引。它还执行逐块候选选择：每个块被赋予其位置中的最大索引分数，并选择分数最高的块。然后，它将所选块覆盖的位置收集到一个比最终 Top-K 集合更大的候选池中。例如，选择 2,048 个块，每个块有 8 个位置，则产生 16,384 个候选位置。该池定义了后续索引器的搜索范围；最终的 Top-K 选择决定了每一层读取哪些主 KV 条目。

This first Full Mode layer scores all causally visible main KV positions and produces the Top-K indices for its own attention. It also performs blockwise candidate selection: each block is assigned the maximum index score among its positions, and the blocks with the highest scores are selected. It then collects the positions covered by the selected blocks into a candidate pool larger than the final Top-K set. For example, selecting 2,048 blocks with 8 positions each yields 16,384 candidate positions. This pool defines where later indexers search; the final Top-K selection determines which main KV entries each layer reads.

Reindex Mode 中的后续层仅对相应查询的候选位置进行评分，并在该池内选择自己的 Top-K 条目。Reuse Mode 中的层不执行新的索引，并使用针对其复用的主 KV 计算的最新 Top-K 索引。因此，候选池在索引层之间共享，而它们的最终选择可以不同。

Subsequent layers in Reindex Mode score only the candidate positions for the corresponding query and select their own Top-K entries within that pool. Layers in Reuse Mode perform no new indexing and use the latest Top-K indices computed against the main KV they reuse. Thus, the candidate pool is shared across indexing layers, while their final selections can differ.

对于固定的候选池大小，每个后续索引器对每个查询评分的的位置数量是有界的，与上下文长度无关。第一个 Full Mode 层仍然扫描整个因果可见范围。因此，分层索引降低了后续索引器评估的开销，同时保留了初始的全范围遍历。

For a fixed candidate-pool size, the number of positions scored per query by each subsequent indexer is bounded independently of context length. The first Full Mode layer still scans the entire causally visible range. Hierarchical indexing therefore reduces the cost of later indexer evaluations while retaining the initial full-range pass.

2.4.1 单遍 mHC 2.4.1 Single-Pass mHC

在 DeepSeek-V4 中，我们引入了 mHC，它在相邻的 Transformer 块之间维护 \(n\) 个残差流。对于每个 token，我们用 \(X_{l} \in \mathbb{R}^{n \times d}\) 表示这些流，其中 \(l\) 是块索引，\(d\) 是隐藏维度。这些流的更新如下：

In DeepSeek-V4, we introduced mHC, which maintains \(n\) residual streams between adjacent Transformer blocks. For each token, we denote these streams by \(X_{l} \in \mathbb{R}^{n \times d}\), where \(l\) is the block index and \(d\) is the hidden dimension. The streams are updated as follows:

其中 \(A_{l} \in \mathbb{R}^{1 \times n}\)、\(C_{l} \in \mathbb{R}^{n \times 1}\) 和 \(B_{l} \in \mathbb{R}^{n \times n}\) 是从 \(X_{l}\) 预测的逐 token 系数。系数预测器 \(\mathcal{H}\) 包括归一化和投影。

where \(A_{l} \in \mathbb{R}^{1 \times n}\), \(C_{l} \in \mathbb{R}^{n \times 1}\) and \(B_{l} \in \mathbb{R}^{n \times n}\) are token-wise coefficients predicted from \(X_{l}\). The coefficient predictor \(\mathcal{H}\) includes normalization and projection.

理想情况下，两个块之间的残差变换是从 \((X_{l-1}, Y_{l-1})\) 到 \((X_{l}, \hat{X}_{l})\) 的单一映射，其中 \(\hat{X}_{l} = A_{l} X_{l}\) 是当前块的输入，\(Y_{l-1} = \mathcal{F}_{l-1}(\hat{X}_{l-1})\) 是前一个块的输出。这样的映射需要 \((n+1)d\) 次读取和 \((n+1)d\) 次写入，从而给出激活内存流量的下界 \((2n+2)d\)。在实践中，DeepSeek-V4 使用 (2) 的多遍实现，由于数据依赖关系，三个内核按顺序执行：

Ideally, the residual transformation between two blocks is a single map from \((X_{l-1}, Y_{l-1})\) to \((X_{l}, \hat{X}_{l})\), where \(\hat{X}_{l} = A_{l} X_{l}\) is the current block input and \(Y_{l-1} = \mathcal{F}_{l-1}(\hat{X}_{l-1})\) is the previous block output. Such a map requires \((n+1)d\) reads and \((n+1)d\) writes, giving a lower bound of \((2n+2)d\) on activation memory traffic. In practice, DeepSeek-V4 uses a multi-pass implementation of (2), with three kernels that execute sequentially due to data dependencies:

这三个内核分别读取 \((n+1)d\)、\(nd\) 和 \(nd\) 个值，总共写入 \((n+1)d\) 个值。包括 \(\mathcal{F}_{l}\) 中的预归一化，总激活内存流量为 \((4n+4)d\)，是下界的两倍。

The three kernels read \((n+1)d\), \(nd\) and \(nd\) values respectively and write \((n+1)d\) in total. Including the pre-norm in \(\mathcal{F}_{l}\), the total activation memory traffic is \((4n+4)d\), twice the lower bound.

在 \(\mathcal{H}\) 中，归一化权重被离线折叠到投影权重中，并且 RMS 除法在投影之后应用。因此，三个阶段中的两个可以共享一次残差遍历：残差更新不需要在隐藏维度上进行归约，因此可以计算 \(X_{l}\) 的每个瓦片，并立即用于累积投影输出和计算 RMS 所需的平方和。输入混合无法融合到这一遍中，因为直到所有隐藏瓦片的归约完成，\(A_{l}\) 才可用。因此，它需要第二次读取 \(X_{l}\)。这第二遍还可以包含输入预归一化。这种两遍实现总共需要 \((3n+2)d\) 次激活读写，与下界相比，多了一次对 \(X_{l}\) 的读取。

In \(\mathcal{H}\), the normalization weights are folded into the projection weights offline, and the RMS division is applied after the projection. Two of the three stages can therefore share one traversal of the residual: the residual update does not require a reduction across the hidden dimension, so each tile of \(X_{l}\) can be computed and immediately used to accumulate the projection outputs and the sum of squares needed to compute the RMS. Input mixing cannot be fused into this pass because \(A_{l}\) is not available until the reduction over all hidden tiles is complete. It therefore requires a second read of \(X_{l}\). This second pass can also incorporate input pre-norm. This two-pass implementation would require \((3n+2)d\) activation reads and writes in total, one additional read of \(X_{l}\) compared with the lower bound.

因此，我们引入了单遍 mHC，它将输入混合系数偏移一个块，即每个块使用前一个块产生的混合系数，从而消除了上述依赖关系：

We therefore introduce Single-Pass mHC, which shifts the input-mixing coefficients by one block, i.e. every block consumes the mixing coefficients produced by the previous one, so that the above dependency disappears:

输入混合现在使用 \(A_{l-1}\) 而不是 \(A_{l}\)，因此它不再依赖于从 \(X_{l}\) 计算出的系数。\(X_{l}\) 的每个分块因此可以立即用于输入混合和系数预测，而无需等待完整的归约。根据经验，这种偏移带来的性能下降可以忽略不计。

Input mixing now uses \(A_{l-1}\) instead of \(A_{l}\), so it no longer depends on the coefficients computed from \(X_{l}\). Each tile of \(X_{l}\) can therefore be used immediately for both input mixing and coefficient prediction, without waiting for the full reduction. Empirically, this shift incurs negligible performance degradation.

对于预训练，我们保留现有的多内核实现，因为偏移只改变了每个块应用的混合系数。对于部署，我们将残差更新、输入混合和系数预测融合到一个单一内核 Mega-mHC 中。该内核实现 mHC 时进行 \((3n+2)d\) 次激活读取和写入，实现单遍 mHC 时进行 \((2n+2)d\) 次激活读取和写入。Mega-mHC 沿隐藏维度分块处理 \(X_{l}\)。每个分块用于计算混合输入，并累积预测下一个块的 \((A_{l},B_{l},C_{l})\) 所需的数量。该内核还集成了输入预归一化和 FP8 转换。因此，残差被读取一次并写入一次，达到了理想映射的 \((n+1)d\) 次读取和 \((n+1)d\) 次写入，并将我们原始实现的激活内存流量减半。

For pre-training, we keep the existing multi-kernel implementation, since the shift only changes which mixing coefficients each block applies. For deployment, we fuse residual update, input mixing, and coefficient prediction into a single kernel, Mega-mHC. The kernel implements mHC with \((3n+2)d\) activation reads and writes and Single-Pass mHC with \((2n+2)d\) activation reads and writes. Mega-mHC processes \(X_{l}\) in tiles along the hidden dimension. Each tile is used to compute the mixed input and to accumulate the quantities needed to predict \((A_{l},B_{l},C_{l})\) for the next block. The kernel also incorporates input pre-norm and FP8 conversion. The residual is thereby read once and written once, attaining the \((n+1)d\) reads and \((n+1)d\) writes of the ideal map and halving the activation memory traffic of our original implementation.

2.4.2 Engram 2.4.2 Engram

我们用 Engram 增强 DeepSeek-V4.1-Flash，Engram 是我们先前工作中引入的条件记忆模块，用于将记忆与计算解耦。我们遵循原始 Engram 设计——分词器压缩、多头哈希、上下文感知门控和多分支集成——并做了两处修改。第一，我们省略了短因果卷积，因为其性能提升不足以证明在我们推理栈中增加复杂性的合理性。第二，我们用基于动量的更新后接 Sinkhorn 平衡来优化 Engram 嵌入，详见第 2.5 节。

We augment DeepSeek-V4.1-Flash with Engram, the conditional memory module introduced in our previous work to decouple memorization from computation. We follow the original Engram design—tokenizer compression, multi-head hashing, context-aware gating, and multi-branch integration—with two modifications. First, we omit the short causal convolution because its performance gains do not justify the added complexity in our inference stack. Second, we optimize the Engram embedding with momentum-based update followed by Sinkhorn balancing, as detailed in Section 2.5.

我们将 196B Engram 参数均匀分配到两个模块。每个模块使用 \(N\)-gram 阶数 \(\{2,3,4\}\)，具有 8 个哈希头，每个阶数的总嵌入维度为 2048。每个头索引一个约 16M 条目的表，表大小选择为不同的素数。嵌入表和键/值投影均使用 FP8 精度。模块放置在层 1 和层 14（零索引）以平衡训练流水线各阶段的内存使用。在推理期间，确定性寻址使得嵌入能够通过后台 RDMA 传输从主机内存预取，第一个模块的预取与第一个 Transformer 块中的计算重叠。Engram 训练和推理的进一步实现细节在第 3.1.3 节讨论。

We allocate 196B Engram parameters evenly across two modules. Each module uses \(N\)-gram orders \(\{2,3,4\}\), with 8 hash heads and a total embedding dimension of 2048 per order. Each head indexes a table of approximately 16M entries, with table sizes chosen to be distinct primes. Both the embedding tables and the key/value projections use FP8 precision. The modules are placed at layers 1 and 14 (zero-indexed) to balance memory usage across training pipeline stages. During inference, deterministic addressing enables embeddings to be prefetched from host memory via background RDMA transfers, with prefetching for the first module overlapping computation in the first Transformer block. Further implementation details for Engram training and inference are discussed in Section 3.1.3.

2.4.3 DSpark 2.4.3 DSpark

我们为 DeepSeek-V4.1-Flash 配备了 DSpark，这是一个推测解码模块，结合了半自回归草稿生成与置信度调度验证。草稿器由三个 Transformer 块组成，滑动注意力窗口大小为 128 个 token。对这些块的一次前向传播并行计算五个草稿位置的基础 logits，同时一个轻量级马尔可夫头对草稿 token 之间的依赖关系进行建模。一个置信度头预测每个位置的条件接受概率，这些概率用于估计前缀存活概率。调度器将这些估计值与分析得到的引擎吞吐量曲线相结合，动态选择每个请求的验证长度，旨在当前系统负载下最大化预期的系统级 token 吞吐量。

We equip DeepSeek-V4.1-Flash with DSpark, a speculative decoding module that combines semi-autoregressive drafting with confidence-scheduled verification. The drafter comprises three Transformer blocks with a sliding attention window of 128 tokens. A single forward pass through these blocks computes base logits for five draft positions in parallel, while a lightweight Markov head models dependencies among the draft tokens. A confidence head predicts per-position conditional acceptance probabilities, which are used to estimate prefix survival probabilities. The scheduler combines these estimates with profiled engine throughput curves to dynamically select the verification length for each request, aiming to maximize expected system-wide token throughput under the current system load.

与 DeepSeek-V3 中的 MTP 模块不同——后者在整个预训练过程中与骨干网络联合训练——DSpark 在预训练之后的一个专门阶段引入。在此阶段，我们仅训练 DSpark，同时保持骨干网络冻结。在后训练期间，我们继续与骨干网络一起训练 DSpark，但不将 DSpark 目标的梯度传播到骨干网络。这使 DSpark 与不断演化的策略保持一致，从而能够加速在线服务以及 RL 和 OPD 的 rollout 生成。

Unlike the MTP module in DeepSeek-V3, which is trained jointly with the backbone throughout pre-training, DSpark is introduced in a dedicated stage after pre-training. In this stage, we train only DSpark while keeping the backbone frozen. During post-training, we continue to train DSpark alongside the backbone, without propagating gradients from the DSpark objective into the backbone. This keeps DSpark aligned with the evolving policy, enabling it to accelerate both online serving and rollout generation for RL and OPD.

2.4.4 FP4 主 KV 缓存 2.4.4 FP4 Main KV Cache

长上下文智能体工作负载需要每个请求的大型 KV 缓存，增加了服务成本。DeepSeek-V4 已对 FP4 索引器查询和键使用量化感知训练（QAT），加速索引计算并减少索引器缓存大小。我们采用 OCP 标准的 MXFP4 格式以支持尽可能多的硬件平台，尽管在我们的实验中替代格式具有更高的准确率。我们现在将 QAT 扩展到主 KV 缓存，其中 FP4 减少存储而非加速矩阵乘法。在注意力机制之前对缓存值进行反量化，使我们能够使用更精确的格式，而无需该格式的原生矩阵乘法支持，从而保持跨硬件平台的兼容性。

Long-context agent workloads require large per-request KV caches, increasing serving costs. DeepSeek-V4 already uses quantization-aware training (QAT) for FP4 indexer queries and keys, accelerating index computation and reducing the indexer cache size. We adopt the OCP-standard MXFP4 format to support as many hardware platforms as possible, despite the higher accuracy of alternative formats in our experiments. We now extend QAT to the main KV cache, where FP4 reduces storage rather than accelerates matrix multiplication. Dequantizing cached values before attention allows us to use a more accurate format without requiring native matrix-multiplication support for that format, preserving compatibility across hardware platforms.

在评估的约四位格式中，我们选择 E2M1，每 16 个通道一个 E4M3 缩放因子，遵循 NVFP4 但省略其第二级全局缩放因子，以平衡准确率和简单性。省略此缩放因子为主 KV 缓存留下了充足的动态范围：该格式支持高达 \(448 \times 6 = 2688\) 的幅度，远高于缓存的幅度界限。在 DeepSeek-V4.1-Flash 中，训练得到的最大 RMSNorm 权重幅度约为 \(1\)。经过 RMS 归一化后，\(512\) 通道 KV 潜变量的 L2 范数至多约为 \(\sqrt{512}\)。RoPE 保持该范数，因此旋转后各通道的最大绝对值也受限于约 \(\sqrt{512} \approx 22.6\)。此外，训练期间观察到的最大幅度约为 \(10\)。因此，省略全局缩放因子不会导致可测量的准确率下降，并简化了缓存布局。

Among the approximately four-bit formats evaluated, we select E2M1 with one E4M3 scale per 16 channels, following NVFP4 but omitting its second-level global scale to balance accuracy and simplicity. Omitting this scale leaves ample dynamic range for the main KV cache: the format supports magnitudes up to \(448 \times 6 = 2688\), far above the cache's magnitude bound. In DeepSeek-V4.1-Flash, the largest trained RMSNorm weight magnitude is approximately \(1\). After RMS normalization, the L2 norm of the \(512\)-channel KV latent is at most approximately \(\sqrt{512}\). RoPE preserves this norm, so the maximum absolute value across channels after rotation is also bounded by approximately \(\sqrt{512} \approx 22.6\). Besides, the maximum magnitude observed during training is around \(10\). Therefore, omitting the global scale causes no measurable decrease in accuracy and simplifies the cache layout.

为了在 DeepSeek-V4.1-Flash 中启用 FP4 主 KV 缓存存储，我们在后训练期间引入 QAT。非 RoPE 和 RoPE 组件使用相同的量化格式。我们在 RoPE 之后量化缓存：在我们的实验中，在 RoPE 之前量化仅带来边际准确率提升，并会在解码期间引入额外开销。由于 SWA KV 缓存对量化敏感，我们对其保留 FP8。与 DeepSeek-V4 中的 FP8 主 KV 缓存相比，此格式几乎将存储占用减半，无论是在 HBM 中还是卸载到 SSD 时。

To enable FP4 main KV cache storage in DeepSeek-V4.1-Flash, we introduce QAT during post-training. The non-RoPE and RoPE components use the same quantization format. We quantize the cache after RoPE: quantizing before RoPE yields only a marginal accuracy improvement in our experiments and would introduce additional overhead during decoding. We retain FP8 for the SWA KV cache due to its sensitivity to quantization. Compared with the FP8 main KV cache in DeepSeek-V4, this format nearly halves the storage footprint, both in HBM and when offloaded to SSD.

2.5 优化 2.5 Optimization

在 DeepSeek-V4 所使用的优化配置基础上，我们做了一些新的修改，以更好地与架构设计对齐。

Building upon the optimization configuration used in DeepSeek-V4, we make some new modifications to better align with the architectural design.

首先，我们使用按头（head-wise）Muon，即在应用 Muon 更新之前，将 Query 权重按头拆分。这里我们简要讨论这种设计的动机。将 Muon 视为预条件梯度下降，原始 Muon 对所有头使用一个预条件器，而按头 Muon 为不同的头提供不同的预条件器。这种设计能更好地处理注意力机制头之间的异质性。因此，我们观察到按头 Muon 优于原始 Muon。按头 Muon 的经验优势也在 GLM 5 和 Kimi-K3 中得到验证。

First, we use head-wise Muon, where Query weights are split by head before applying the Muon update. Here, we briefly discuss the motivation for such a design. By viewing Muon as a preconditioned gradient descent, vanilla Muon uses one preconditioner for all heads, whereas head-wise Muon provides different preconditioners for different heads. This design can better handle the heterogeneity across attention heads. As a result, we observe that head-wise Muon outperforms vanilla Muon. The empirical advantage of head-wise Muon is also validated in GLM 5 and Kimi-K3.

其次，将 Adam 应用于新引入的 Engram 参数会大幅增加优化器状态的内存占用。为了减少训练期间的内存使用，我们改为使用基于动量的更新后接 Sinkhorn 平衡来优化 Engram 嵌入表、词元嵌入和预测头。Sinkhorn 平衡此前已在 SinkGD 中应用于线性层权重矩阵；这里我们将其扩展到这些大型参数矩阵。与 Muon 类似，这种方法只需要一个动量缓冲区，同时在经验上优于 Adam。

Second, applying Adam to the newly introduced Engram parameters substantially increases the optimizer-state memory footprint. To reduce memory usage during training, we instead optimize the Engram embedding tables, token embedding, and prediction head using a momentum-based update followed by Sinkhorn balancing. Sinkhorn balancing has previously been applied to linear-layer weight matrices in SinkGD; here, we extend it to these large parameter matrices. Like Muon, this approach requires only a momentum buffer while empirically outperforming Adam.

我们对归一化层权重和其他非矩阵参数（包括偏置和缩放因子）保留 AdamW。我们对语言模型主干中的线性变换权重矩阵、Engram 投影层和视觉-语言投影器使用 Muon。我们对 Query 和 Key 权重使用按头 Muon。我们对 Muon 应用解耦权重衰减和 Nesterov 动量；归一化层权重也受权重衰减影响，而偏置和缩放因子则不受影响。Sinkhorn 平衡更新也使用 Nesterov 动量，但不应用权重衰减。在预训练期间，我们保持视觉编码器冻结，直到学习率衰减阶段，而其最终归一化层和视觉-语言投影器保持可训练。在学习率衰减开始时，我们解冻视觉编码器，并以较小的学习率与 LLM 联合优化。

We retain AdamW for normalization-layer weights and other non-matrix parameters, including biases and scaling factors. We use Muon for the weight matrices of linear transformations in the language-model backbone, the Engram projection layers, and the vision-language projector. We use head-wise Muon for Query and Key weights. We apply decoupled weight decay and Nesterov momentum to Muon; normalization-layer weights are also subject to weight decay, whereas biases and scaling factors are not. The Sinkhorn-balanced update also uses Nesterov momentum but does not apply weight decay. During pre-training, we keep the vision encoder frozen until the learning-rate decay stage, while its final normalization layer and the vision–language projector remain trainable. At the onset of learning-rate decay, we unfreeze the vision encoder and optimize it jointly with the LLM with a smaller learning rate.

完整过程总结在算法 1 中。在高层次上，它遵循与 Muon 相同的工作流程，用 Sinkhorn 平衡代替 Newton–Schulz 正交化。我们用 \(m\) 表示较大的矩阵维度，它对应于嵌入表和预测头的词汇量大小，并用 \(n\) 表示隐藏维度。

The complete procedure is summarized in Algorithm 1. At a high level, it follows the same workflow as Muon, with Sinkhorn balancing taking the place of Newton–Schulz orthogonalization. We denote the larger matrix dimension by \(m\), which corresponds to the vocabulary size for embedding tables and prediction heads, and denote the hidden dimension by \(n\).

给定 Nesterov 动量更新 \(\widehat{G}_{t}\)，Sinkhorn 平衡寻找对角缩放矩阵 \(D_{r}\) 和 \(D_{c}\)，使得

Given the Nesterov momentum update \(\widehat{G}_{t}\), Sinkhorn balancing finds diagonal scaling matrices \(D_{r}\) and \(D_{c}\) such that

因此，该过程近似地均衡更新矩阵的行和列的 RMS。这里，一行对应一个 token 索引或 n-gram 标识；一列编码一个隐藏特征。Sinkhorn 平衡通过沿行和列进行归一化来利用这种 token-特征结构。为了数值稳定性，满足 \(\rho_{i}\leqslant\tau\bar{\rho}\) 的行被掩码。因子 \(\sqrt{n}\) 将单位行 \(\ell_{2}\) 范数转换为单位行 RMS。另外，我们调整有效学习率为 \(\widetilde{\eta}_{t}=\gamma\eta_{t}\) 以匹配 Adam 的更新幅度。我们设置 \(\gamma=0.18\)，这接近 Moonlight 中使用的因子 \(0.2\)。

Thus, the procedure approximately equalizes the row-wise and column-wise RMS of the update matrix. Here, one row corresponds to one token index or n-gram identity; and one column encodes one hidden feature. Sinkhorn balancing exploits this token–feature structure by normalizing along both rows and columns. For numerical stability, rows satisfying \(\rho_{i}\leqslant\tau\bar{\rho}\) are masked. The factor \(\sqrt{n}\) converts unit row \(\ell_{2}\) norm into unit row-wise RMS. Separately, we adjust the effective learning rate as \(\widetilde{\eta}_{t}=\gamma\eta_{t}\) to match the update magnitude of Adam. We set \(\gamma=0.18\), which is close to the factor \(0.2\) used in Moonlight.

更广泛地说，Sinkhorn 平衡与利用矩阵或张量轴结构的优化器密切相关。例如，Adafactor 以不同方式进行行和列归一化，而 Adam-mini 对嵌入表和预测头使用另一种行归一化。这些归一化策略可能在优化性能和通信开销上有所不同。我们将更详细的研究留作未来方向。

More broadly, Sinkhorn balancing is closely related to optimizers that exploit matrix or tensor axis structure. For example, Adafactor conducts row- and column-wise normalization in a different manner, and Adam-mini uses an alternative row-wise normalization for embedding tables and prediction head. These normalization strategies may differ in optimization performance and communication overhead. We leave more detailed investigation as a future direction.

3.1.1 多模态训练基础设施 3.1.1 Multimodal Training Infrastructure

视觉编码器首先使用对比目标进行优化，然后使用生成式下一个词预测损失进行微调。在对比阶段，损失是在完整的文本和视觉对批次上计算的，因此两种模态的特征必须在数据并行秩之间进行全收集，这会产生大量的通信开销。由于文本特征的梯度仅依赖于收集到的视觉特征——对称地，视觉特征的梯度仅依赖于收集到的文本特征——每个全收集操作可以与前向或后向传递重叠，而不是阻塞流水线：

The vision encoder is first optimized with a contrastive objective before being fine-tuned with a generative next-token prediction loss. In the contrastive phase, the loss is computed over a full batch of text and vision pairs, so the features of both modalities must be all-gathered across data-parallel ranks, incurring substantial communication. Because the gradient of the text features depends only on the gathered visual features—and, symmetrically, the gradient of the visual features depends only on the gathered text features—each all-gather can be overlapped with the forward or backward pass instead of stalling the pipeline:

其中 \(V\) 和 \(T\) 分别表示视觉和文本特征，\((A \| C)\) 表示计算 \(A\) 与通信 \(C\) 的重叠，\(\nabla\) 表示梯度计算。在此调度中，视觉特征在文本前向传递期间收集，文本特征在文本后向传递期间收集，因此两个全收集操作都完全隐藏在有用的计算之后。

where \(V\) and \(T\) denote the visual and text features, \((A \| C)\) denotes the overlap of computation \(A\) with communication \(C\), and \(\nabla\) denotes the gradient computation. In this schedule, the visual features are gathered during the text forward pass and the text features during the text backward pass, so that both all-gathers are hidden entirely behind useful computation.

为了处理视觉编码器和 LLM 之间的模型和数据异构性，我们采用了近期训练系统中使用的解耦编码器设计。视觉编码器被复制到 LLM 参数树之外，每个训练步骤分为三个阶段：视觉编码器前向、LLM 前向/后向和视觉编码器后向。这种分离防止了视觉编码器和 LLM 计算之间的干扰。负载均衡的视觉处理仅限于第一阶段和最后阶段，而 LLM 阶段则没有视觉计算，并保留了纯文本训练的并行策略。

To handle the model and data heterogeneity between the vision encoder and the LLM, we adopt the disaggregated encoder design used in recent training systems. The vision encoder is replicated outside the LLM parameter tree, and each training step is divided into three phases: vision encoder forward, LLM forward/backward, and vision encoder backward. This separation prevents interference between vision encoder and LLM computation. Load-balanced vision processing is confined to the first and last phases, while the LLM phase remains free of vision computation and preserves the parallel strategy of text-only training.

DeepSeek-V4.1-Flash 在长达一百万个 token 的序列上进行训练，在预训练和后训练阶段，很大一部分训练发生在超长序列长度上。在这些序列长度下，多模态样本会造成严重的 I/O、CPU 和内存瓶颈。

DeepSeek-V4.1-Flash is trained on sequences of up to one million tokens, with a substantial share of training occurring at ultra-long sequence lengths in both the pre-training and post-training stages. At these sequence lengths, multimodal samples create heavy I/O, CPU, and memory bottlenecks.

平衡图像分片。在预训练期间，单个超长、图像密集的序列在加载时可能耗尽一个主机的 I/O、CPU 和内存，因此每个序列的图像在 CP 秩之间进行负载均衡分片，并且每个图像仅加载一次。由于图像只读取一次，加载始终隐藏在计算之后，只要

Balanced image sharding. During pre-training, a single ultra-long, image-dense sequence can exhaust one host’s I/O, CPU, and memory during loading, so the images of each sequence are sharded across the CP ranks with load balancing, and each image is loaded exactly once. With images read once, loading stays hidden behind compute whenever

其中 \(N\) 是 token 数量，\(\rho\) 是每个 token 的原始字节数，\(C\) 是每个 token 的计算量，而 \(B_{\mathrm{IO}}\)、\(B_{\mathrm{GPU}}\) 分别是文件系统和 GPU 带宽。由于 \(N\) 被约去，该准则仅涉及每个 token 的量（\(\rho\) 和 \(C\)），与序列长度和集群规模无关；\(\rho\) 由视觉模块配置决定（例如分辨率上限或空间下采样）。因此，存储吞吐量仅对每个 token 计算量低的小模型（如消融实验中的模型）成为瓶颈，而生产规模的模型仍受计算量限制。

where \(N\) is the token count, \(\rho\) the raw bytes per token, \(C\) the per-token compute, and \(B_{\mathrm{IO}}\), \(B_{\mathrm{GPU}}\) the file-system and GPU bandwidths. Since \(N\) cancels, the criterion involves only per-token quantities (\(\rho\) and \(C\)), independent of sequence length and cluster size; \(\rho\) is set by the vision-module configuration (e.g., the resolution cap or the spatial downsample). Storage throughput therefore becomes a bottleneck only for small models with low per-token compute, as in ablations, while production-scale models remain compute-bound.

增量图像传输。除了上述均衡分片外，强化学习 rollout 仅以增量方式将图像传输到推理引擎，并将引擎在 CPU 侧的解码和预处理输出缓存到分布式文件系统上，以便在多次 rollout 和后续训练中复用。

Incremental image transfer. Besides the balanced sharding above, the reinforcement-learning rollout transfers images to the inference engine only incrementally, and caches the engine’s CPU-side decoding and preprocessing outputs on a distributed file system for reuse across rollouts and subsequent training.

3.1.2 CSA2 的注意力共享训练 3.1.2 Attention Sharing Training for CSA2

在第 2.3 节中，我们介绍了 CSA2，一种具有三种模式的注意力共享方法，其中一些模式涉及跨多个层共享主 KV、索引器 K 和 Top-K 索引中的一个或多个。在大规模分布式训练中支持 CSA2 需要在注意力计算本身之外进行额外的协调。特别是，共享注意力组件的层可能被放置在不同的流水线阶段，这使得直接的模块复用与传统的阶段局部执行不兼容。因此，我们采用了几种设计来支持 CSA2 训练。

In Section 2.3, we introduce CSA2, an attention sharing method with three modes, some of which involve sharing one or more of the main KV, the indexer K, and the Top-K indices across multiple layers. Supporting CSA2 in large-scale distributed training requires additional coordination beyond the attention computation itself. In particular, layers that share attention components may be placed on different pipeline stages, making direct module reuse incompatible with conventional stage-local execution. Therefore, we adopt several designs to support CSA2 training.

影子索引器通过在参与共享的每个阶段放置一个轻量级的可执行副本，同时为共享参数保留单一的逻辑所有者，来解决这一问题。所有者仍然负责优化和检查点保存，而参数同步和梯度聚合则使影子副本在整个训练过程中保持一致。这种设计保留了原始模型语义，而无需流水线调度器将共享层视为特殊的执行单元。

Shadow indexers address this issue by placing a lightweight executable replica on each participating stage while retaining a single logical owner for the shared parameters. The owner remains responsible for optimization and checkpointing, whereas parameter synchronization and gradient aggregation keep the shadow replicas consistent throughout training. This design preserves the original model semantics without requiring the pipeline scheduler to treat shared layers as a special execution unit.

流水线载荷扩展提供了当源层和消费层跨越流水线边界时下游消费者所需的中间表示和稀疏路由信息。这些状态被纳入现有的点对点通信路径，并与上下文并行一致地进行分区，从而在保持相应梯度流的同时避免不必要的复制。

Pipeline payload extensions provide the intermediate representations and sparse routing information required by downstream consumers when the source and consumer layers cross a pipeline boundary. These states are incorporated into the existing point-to-point communication path and partitioned consistently with context parallelism, avoiding unnecessary replication while maintaining the corresponding gradient flow.

微批次级共享状态管理跟踪与并发活跃的流水线微批次相关的状态，并协调它们在前向执行、激活重计算和反向传播过程中的生命周期。共享状态会保留到其最终消费者完成为止，然后被及时释放以限制额外的内存开销。相同的运行时抽象还处理阶段放置和源-消费者关系，使得注意力实现能够访问共享状态，而不依赖于物理流水线布局。

Micro-batch-level shared-state management tracks the states associated with concurrently active pipeline micro-batches and coordinates their lifetimes across forward execution, activation recomputation, and backward propagation. Shared states are retained until their final consumer has completed and are then released promptly to limit additional memory overhead. The same runtime abstraction also handles stage placement and source–consumer relationships, allowing the attention implementation to access shared states without depending on the physical pipeline layout.

结合对优化器、检查点保存、预热和计算图追踪工作流的轻量级适配，这些机制使得 CSA2 能够在现有的分布式训练接口和流水线调度下透明地运行。

Together with lightweight adaptations to the optimizer, checkpointing, warm-up, and computation-graph tracing workflows, these mechanisms enable CSA2 to operate transparently under the existing distributed training interface and pipeline schedules.

3.1.3 Engram 3.1.3 Engram

Engram 嵌入表按行划分，分布在由 engram 并行度指定的专用进程组上。组大小控制着每设备内存使用量与嵌入查找通信范围之间的权衡。优化器状态进一步分片存储在每个表分区的副本上。Engram 查找索引仅依赖于输入 token 序列。因此，在每个流水线阶段开始处理当前训练步骤的微批次之前，会为整个本地批次启动嵌入预取，从而最小化对流水线执行的干扰。嵌入梯度在反向传播期间被缓冲，并在主干反向传播后返回给其所属的 rank。为了与多模态训练高效集成，嵌入预取和梯度传输被调度为与视觉编码器的前向和反向计算重叠。嵌入以 FP8 格式存储和获取，检索到的值和缩放因子直接传递给后续的 GEMM。对于 Engram 表更新，Sinkhorn 归一化在迭代间维护行和列缩放向量，以避免重复写入完整的归一化矩阵。行归一化和部分列统计的累积被融合到单个内核中，以进一步减少内存流量。在 RL 推演期间，Engram 嵌入表常驻 GPU 内存。这种放置减少了主机内存压力，并有助于避免由主机内存碎片引起的内存不足故障。

Engram embedding tables are partitioned by row across dedicated process groups of engram parallel size. The group size controls the trade-off between per-device memory usage and the communication scope of embedding lookups. Optimizer states are further sharded across replicas of each table partition. Engram lookup indices depend solely on the input token sequence. Embedding prefetch is therefore initiated for the entire local batch before each pipeline stage begins processing microbatches for the current training step, minimizing interference with pipeline execution. Embedding gradients are buffered during backward and returned to their owning ranks after the backbone backward pass. For efficient integration with multimodal training, embedding prefetch and gradient transfers are scheduled to overlap with the vision encoder’s forward and backward computation. Embeddings are stored and fetched in FP8, with the retrieved values and scaling factors passed directly to the following GEMM. For Engram table updates, Sinkhorn normalization maintains row and column scaling vectors across iterations to avoid repeated writes of the full normalized matrix. Row normalization and the accumulation of partial column statistics are fused into a single kernel to further reduce memory traffic. During RL rollouts, Engram embedding tables remain resident in GPU memory. This placement reduces host memory pressure and helps avoid out-of-memory failures caused by host memory fragmentation.

3.2 推理系统 3.2 Inference System

DeepSeek-V4.1-Flash 在设计时将推理效率作为首要考量。尽管其架构在概念上较为复杂，但由此产生的推理内核流程却极为简洁。通过合理的内核融合，我们将复杂的操作封装起来，并在少量融合内核中保持硬件资源完全流水线化——包括 FlashMLA 中的融合 RoPE-注意力机制-RoPE-cast 内核，DeepGEMM 中的 Mega-Gate、Mega-mHC 和 Mega-MoE 内核，TileKernels 中的内核，以及 DeepSelect 中的 TopK 内核。因此，绝大多数 Transformer 层——即那些 CSA2 以复用模式运行的层——在预填充阶段仅执行 15 个内核，在解码阶段仅执行 11 个内核，从而实现了高吞吐量和低延迟的推理。

DeepSeek-V4.1-Flash is designed with inference efficiency as a first-class concern. Although its architecture is conceptually complex, the resulting inference kernel flow is remarkably concise. Through reasonable kernel fusion, we encapsulate the intricate operations and keep hardware resources fully pipelined inside a small number of fused kernels—including the fused-RoPE-attention-RoPE-cast kernel in FlashMLA, the Mega-Gate, Mega-mHC, and Mega-MoE kernels in DeepGEMM, the kernels in TileKernels, and the TopK kernel in DeepSelect. As a result, the vast majority of Transformer layers—those whose CSA2 operates in Reuse Mode—execute with only 15 kernels during prefill and 11 during decode, thereby achieving both high-throughput and low-latency inference.

在部署层面，我们采用编码器-预填充-解码（EPD）解耦，使视觉编码、预填充和解码能够独立扩展并重叠执行。

At the deployment level, we adopt Encoder–Prefill–Decode (EPD) disaggregation, enabling vision encoding, prefill, and decoding to scale independently and overlap in execution.

3.2.1 持久化 KV 缓存管理 3.2.1 Persistent KV Cache Management

在相同工作负载下，V4.1 将持久化 KV 缓存的占用降至 V4 的 1/8。这一降低由两个相乘的因素构成：持久化 KV 缓存不再存储 SWA KV，这几乎使其大小减半；同时，通过架构和精度优化，其中保留的全局 KV 进一步压缩至 V4 占用的 1/4。

Under identical workloads, V4.1 reduces the persistent KV cache footprint to 1/8 of that of V4. Two multiplicative factors account for this reduction: the persistent KV cache no longer stores SWA KV, which almost halves its size, and the global KV retained in it is further compressed to 1/4 of V4's footprint through architectural and precision optimizations.

在 V4 部署中，SWA KV 占持久化 KV 缓存容量的近一半。在该缓存内，全局 KV 和 SWA KV 独立管理，遵循 LRU 淘汰策略。全局 KV 完整存储，命中时可复用完整前缀。相比之下，SWA KV 缓存在两个特定点——提示末尾和输出末尾——以促进重新生成和多轮会话；命中时允许从该缓存位置恢复计算。为保持高命中率，我们在 SSD 上配置了足够大的持久化 KV 缓存，使得在典型工作负载下，两种 KV 均驻留超过 72 小时。尽管仅在指定位置保留 \(n_{\mathrm{win}}\) 个 KV 条目，未压缩的 SWA KV 缓存仍会产生大量存储开销，尤其是在短轮次的多轮对话中。

In the V4 deployment, SWA KV accounts for nearly half of the persistent KV cache capacity. Within this cache, global KV and SWA KV are managed independently, governed by an LRU eviction policy. Global KV is stored in its entirety, and upon a hit, the complete prefix is reused. In contrast, SWA KV is cached at two specific points—the end of the prompt and the end of the output—to facilitate regeneration and multi-turn sessions; a hit allows computation to resume from that cached position. To maintain a high hit rate, we configured a sufficiently large persistent KV cache on SSD such that, under typical workloads, both types of KV remain resident for over 72 hours. Despite retaining only \(n_{\mathrm{win}}\) KV entries at designated positions, the uncompressed SWA KV cache still incurs a substantial storage overhead, especially in multi-turn conversations with short turns.

持久化存储 SWA KV 既昂贵又低效，因为其访问模式与持久化 KV 缓存的长期保留策略不匹配。与表现出长尾复用的全局 KV 不同，SWA KV 仅在活动会话内一个狭窄的分钟级窗口内被复用，一旦会话结束或下一轮开始即失效。V4 技术报告提出了 Zero SWA Caching，通过重新计算缺失的 SWA KV 来避免存储开销。然而，精确恢复需要对 \(L \times n_{\mathrm{win}}\) 个 token 进行完整前向传播，其成本在生产部署中被证明过高。

Persistently storing SWA KV is both costly and ineffective, because its access pattern does not match the persistent KV cache's long retention policy. Unlike global KV, which exhibits long-tail reuse, SWA KV is reused only within a narrow, minute-scale window inside an active session and becomes dead once the session ends or the next turn begins. The V4 technical report proposed Zero SWA Caching, which avoids the storage overhead by recomputing missing SWA KV. Exact recovery, however, requires a full forward pass over \(L \times n_{\mathrm{win}}\) tokens, whose cost proved prohibitive in production deployments.

因此，V4.1 对持久化 KV 缓存管理进行了如下修订：

V4.1 therefore revises persistent KV cache management as follows:

SWA KV 不再缓存在持久化 KV 缓存中，而是存储在分布式内存池中，该内存池由每台机器主机 DRAM 的 10% 提供。尽管该池的总容量小得多，但其较短的 TTL（仅几分钟）允许过期条目立即回收用于新会话；在真实工作负载下，这种高周转率足以服务绝大多数并发活动会话。全局 KV 仍保留在持久化 KV 缓存中，保证生命周期至少 72 小时。

SWA KV is no longer cached in the persistent KV cache and instead stored in a distributed memory pool provisioned from 10% of the host DRAM on each machine. Although this pool is far smaller in aggregate capacity, its short TTL (only minutes) allows expired entries to be recycled immediately for new sessions; under real-world workloads, this high turnover suffices to serve the vast majority of concurrent active sessions. Global KV remains in the persistent KV cache with a guaranteed lifetime of at least 72 hours.

驱逐 SWA KV 不可避免地会导致未命中，但得益于一种轻量级的回退机制——Encoder SWA 有界重放（详见第 3.2.2 节），这种未命中的代价仍可承受。对于那些不可避免但频率较低的、命中全局 KV 却未命中 SWA KV 的请求，该机制通过仅重算 \(n_{\mathrm{win}}\) 个 token 来恢复缺失状态，而非执行完整的 \(L \times n_{\mathrm{win}}\) 个 token 的前向传播。这种有界重放是该设计的基石：它将一次灾难性的未命中转化为一种优雅且代价低廉的降级，从而为从持久化 KV 缓存中移除 SWA KV 提供了依据。

Evicting SWA KV inevitably causes misses, which stay affordable thanks to a lightweight fallback, Encoder SWA Bounded Replay (detailed in Section 3.2.2). For the inevitable but infrequent requests that hit global KV but miss SWA KV, it recovers the missing state by recomputing only \(n_{\mathrm{win}}\) tokens instead of a full \(L \times n_{\mathrm{win}}\)-token forward pass. This bounded replay is the cornerstone of the design: it turns a catastrophic miss into a graceful, inexpensive degradation, thereby justifying the removal of SWA KV from the persistent KV cache.

3.2.2 SWA 有界重放 3.2.2 SWA Bounded Replay

由于 SWA 依赖跨层累积，精确重建 \(L\) 层的 SWA KV 需要重放 \(L \times n_{\mathrm{win}}\) 个 token。SWA 有界重放则仅重放最近的 \(n_{\mathrm{win}}\) 个 token，并将 SWA 截断至重放片段，接受近似状态：对于从位置 \(s\) 开始的重放，位置 \(i\) 的查询关注 SWA 键的范围为 \([\max(s, i-W+1), i]\)。

Since SWA dependencies accumulate across layers, exactly reconstructing the SWA KV of \(L\) layers would require replaying \(L \times n_{\mathrm{win}}\) tokens. SWA Bounded Replay instead replays only the most recent \(n_{\mathrm{win}}\) tokens and truncates SWA to the replay segment, accepting approximate states: for a replay starting at position \(s\), a query at position \(i\) attends to SWA keys in \([\max(s, i-W+1), i]\).

编码器 SWA 有界重放使前缀缓存仅依赖于全局 KV，从而允许从持久 KV 缓存中移除 SWA KV。

Encoder SWA Bounded Replay makes prefix caching depend only on global KV, allowing SWA KV to be removed from the persistent KV cache.

当编码器 SWA KV 缺失时，我们重放缓存前缀的最后 \(n_{\mathrm{win}}\) 个 token，并将其与未缓存的后续部分一起处理。重放的 token 仅重新生成 SWA KV，复用缓存的全局 KV 而无需重新计算或覆盖，而未缓存的后续部分则同时生成全局 KV 和 SWA KV。

When the encoder SWA KV is missing, we replay the last \(n_{\mathrm{win}}\) tokens of the cached prefix and process them together with the uncached suffix. The replayed tokens regenerate only SWA KV, reusing the cached global KV without recomputation or overwriting, while the uncached suffix generates both global KV and SWA KV.

根据设计，重放的前缀状态是近似的，因此为未缓存后续部分计算的全局 KV 和 SWA KV 依赖于缓存命中位置，在不同位置之间并非数学上完全相同。令人鼓舞的是，我们的实验证据证实，这种有界重放策略几乎不会损害响应质量。

By design, the replayed prefix state is approximate, so the global KV and SWA KV computed for the uncached suffix depend on the cache-hit position and are not mathematically identical across positions. Encouragingly, our experimental evidence confirms that this bounded replay strategy barely compromises response quality.

解码器 SWA 有界重放将解码器前向传播限制在 \(n_{\mathrm{win}}\) 个 token 内，几乎将总预填充计算量减半。

Decoder SWA Bounded Replay bounds the decoder forward pass to \(n_{\mathrm{win}}\) tokens, nearly halving total prefill computation.

在 CED 下，解码器全局 KV 是从最终的编码器隐藏状态投影得到的。在编码器处结束预填充的唯一障碍是解码器 SWA KV，它由每个解码器层自身的隐藏状态生成，并且是前几个解码步骤所需要的。由于我们从不缓存解码器 SWA KV，要精确重建它，需要在最后 \(\frac{L}{2}\times n_{\mathrm{win}}\) 个提示词元上运行 \(\frac{L}{2}\) 个解码器层，当短的未缓存后缀跟在长的已缓存前缀之后时，这种开销很大。因此，我们也将有界重放策略应用于此场景：在每次预填充时，我们重放提示的最后 \(n_{\mathrm{win}}\) 个词元，在相同的 SWA 截断下将其编码器输出馈入解码器层，并且仅将得到的解码器 SWA KV 用于解码，而不用于前缀缓存。

Under CED, decoder global KV is projected from the final encoder hidden states. The only obstacle to ending prefill at the encoder is decoder SWA KV, which is generated from each decoder layer’s own hidden states and is needed by the first decode steps. Since we never cache decoder SWA KV, exactly reconstructing it requires running the \(\frac{L}{2}\) decoder layers over the last \(\frac{L}{2}\times n_{\mathrm{win}}\) prompt tokens, which is expensive when a short uncached suffix follows a long cached prefix. Therefore, we also apply the bounded replay strategy to this scenario: at every prefill, we replay the last \(n_{\mathrm{win}}\) tokens of the prompt, feed their encoder outputs through the decoder layers under the same SWA truncation, and use the resulting decoder SWA KV only for decoding, not for prefix caching.

按照设计，重建的解码器 SWA KV 在数学上并不等同于完整解码器前向传播得到的 KV。此外，我们发现该策略对响应质量的影响可以忽略不计。为了增加安全性，我们在后训练期间额外模拟相同的重放，以实现训练感知的适应。

By design, the reconstructed decoder SWA KV is not mathematically equivalent to that from a full decoder forward pass. Also, we find that this strategy has only a negligible impact on response quality. For added safety, we additionally simulate the same replay during post-training for train-aware adaptation.

4.1 数据构建 4.1 Data Construction

为了追求更高的智能，我们超越了小规模数据实验所反映的通用、样本级质量，更关注提供独特信息增益的多样化语料库之间的整体交互。我们采用更系统化、标准化的数据构建流程，以提高数据质量并优化数据混合。具体而言，基于更全面的评估，我们精心设计了模型参数和训练数据的缩放阶梯，以指导大规模训练运行。我们过滤掉信息增益有限的模型生成内容，包括能力较弱模型的输出和低质量的机器翻译文本。我们将此类内容视为隐式重复，因为它在很大程度上重新表述了现有信息，并且在长时间训练中可能变得有害。我们还探索了模型在环数据迭代方法，作为未来大规模合成数据的基础。此外，我们引入更多领域专家来构建细粒度的数据质量评估维度。与上一版本相比，新语料库纳入了来自新发布的开源仓库、提交、库和新兴框架的更新代码，以覆盖更广泛的编程语言，并更好地反映当代真实世界的软件工程场景。

In pursuit of higher intelligence, we go beyond the general, sample-level quality reflected by small-scale data experiments and focus more on the holistic interactions among diverse corpora that offer unique information gains. We adopt a more systematic and standardized data construction pipeline to improve data quality and optimize the data mixture. Specifically, based on more comprehensive evaluations, a scaling ladder over model parameters and training data is carefully designed to guide large-scale training runs. We filter out model-generated content with limited information gain, including outputs from less capable models and low-quality machine-translated text. We regard such content as implicit duplication, as it largely reformulates existing information and may become detrimental over long training horizons. We also explore model-in-the-loop data iteration approaches as a foundation for future large-scale synthetic data. In addition, we involve more domain experts to construct fine-grained data quality evaluation dimensions. Compared with the previous version, the new corpus incorporates more recent code from newly released open-source repositories, commits, libraries, and emerging frameworks to cover a broader range of programming languages and better reflects contemporary real-world software engineering scenarios.

我们的多模态预训练数据集主要包含三类数据：图文对、交错图文数据和领域特定数据。基于原始网络数据自然提供丰富多模态知识的前提，我们避免大规模数据合成；相反，我们优先以原生形式清洗和利用数据，以在预训练期间实现最直接且可扩展的视觉知识压缩。在初始数据收集期间，我们发现爬取系统过于偏向以文本为中心的网页内容；因此，我们从 Common Crawl 重新引导它，以提高对多模态来源的覆盖。对于图文数据，我们从网页中提取图像及其关联的替代文本，通过应用图文相关性阈值进行过滤，并基于图像语义进行去重。对于交错数据，我们主要从网页和 PDF 构建此子集。处理大规模多模态语料库通常比处理纯文本语料库产生更高的 CPU 和磁盘存储成本，这促使我们将交错数据构建组织为逐步更昂贵的阶段。在图像检索之前，我们应用启发式和统计过滤、去重和质量模型来选择高价值文档。幸存的文档随后被组装成交错图像-文本序列，其中以图像感知的方式再次应用过滤和去重。最后，我们使用 SmolVLM 对图文内容进行严格的质量评分，从而提取高质量的交错数据。在此过程中被过滤掉的文档部分通过筛选和重组回收为额外的图文对。为了弥补网络收集数据的固有局限性，我们还纳入领域特定数据集，以提升模型在细粒度视觉感知（例如，视觉定位和指向）、光学字符识别（OCR）以及长尾知识获取方面的能力。我们还收集了大量的图像-代码对和计算机使用轨迹，以提高多模态智能体式理解。

Our multimodal pre-training dataset primarily comprises three types of data: image-text pairs, interleaved image-text data, and domain-specific data. Operating on the premise that raw web data naturally provides rich multimodal knowledge, we refrained from large-scale data synthesis; instead, we prioritized cleaning and utilizing the data in its native form to achieve the most direct and scalable visual knowledge compression during pre-training. During the initial data collection, we found that our crawling system was overly biased toward text-centric web content; we therefore re-bootstrapped it from Common Crawl to improve its coverage of multimodal sources. For image-text data, we extract images together with their associated alt text from webpages, filter them by applying an image-text relevance threshold, and deduplicate them based on image semantics. For interleaved data, we build this subset predominantly from webpages and PDFs. Processing large-scale multimodal corpora usually incurs higher CPU and disk storage costs than processing text-only corpora, which motivated us to organize the interleaved-data construction into progressively more expensive stages. Before image retrieval, we apply heuristic and statistical filtering, deduplication, and quality models to select high-value documents. The surviving documents are then assembled into interleaved image–text sequences, where filtering and deduplication are applied again in an image-aware manner. Finally, we employ SmolVLM to conduct strict quality scoring on the image-text content, thereby extracting high-quality interleaved data. Documents filtered out during this process are partly recycled into additional image-text pairs via screening and recombination. To compensate for the inherent limitations of web-gathered data, we also incorporate domain-specific datasets to boost the model’s capabilities in fine-grained visual perception (e.g., visual grounding and pointing), optical character recognition (OCR), and the acquisition of long-tail knowledge. We also collect extensive image-code pairs and computer-use trajectories to improve multimodal agentic understanding.

由于我们的纯文本和多模态数据通过不同的流程处理，我们将最终训练语料库构建为两个数据源的并集。对于重叠样本，我们用多模态版本替换纯文本版本，并使用两种配置中较大的轮数。经过此替换后，生成的语料库使用纯文本与多模态数据 \(7{:}1\) 的 token 比例。我们通过联合预取和分配训练样本，在预训练和上下文扩展期间最小化样本重叠。超长文档在混合前被确定性地预分割，以确保训练 token 在数据分片和训练步骤中均匀分布。我们进一步增强了最佳拟合打包算法，实现了至多 \(10^{-4}\) 的填充率。

As our text-only and multimodal data were processed through distinct pipelines, we constructed the final training corpus as the union of both data sources. For overlapping samples, we replace the text-only versions with their multimodal counterparts and use the larger epoch count of the two configurations. After this substitution, the resulting corpus uses a \(7{:}1\) token ratio of text-only to multimodal data. We minimize sample overlap during pre-training and context extension by jointly prefetching and assigning training samples. Ultra-long documents are deterministically pre-split before mixing to ensure a uniform distribution of training tokens across data shards and training steps. We further enhance our best-fit packing algorithm, achieving a padding rate of at most \(10^{-4}\).

4.2.1 模型设置 4.2.1 Model Setups

我们将 Transformer 层数设为 40，隐藏维度 \(d\) 设为 5120。我们采用因果编码器-解码器架构，编码器 20 层，解码器 20 层。前两层使用纯滑动窗口注意力机制。其余 18 个编码器层使用压缩率 \(m=2\) 的 CSA2。这些层被分为三组，每组六层，配置相同。每组中，第一层以 Full Mode 运行，其余五层以 Reuse Mode 运行。20 个解码器层使用压缩率 \(m=1\) 的 CSA2。这些层被分为五组，每组四层。第一组中，第一层以 Full Mode 运行，其余三层以 Reuse Mode 运行。其余四组配置相同：第一层以 Reindex Mode 运行，其余三层以 Reuse Mode 运行。对于所有 CSA2 层，我们将索引器查询头数设为 32，索引器头维度设为 128，稀疏注意力（即注意力 top-k）选择的 KV 条目数设为 512。我们将查询头数设为 64，头维度设为 512，查询压缩维度设为 1280。对于分层稀疏索引器，我们最多选择 2,048 个块，每个块 8 个位置，总共最多产生 16,384 个候选位置。输出投影组数设为 8，每个中间注意力输出的维度设为 1024。对于滑动窗口注意力的额外分支，窗口大小 \(n_{\mathrm{win}}\) 设为 128。我们在所有 Transformer 块中使用 MoE 层，采用 SwiGLU 激活函数，阈值截断为 10。每个 MoE 层包含 1 个共享专家和 384 个路由专家，每个专家的中间隐藏维度为 2304。在路由专家中，每个 token 将激活 6 个专家。对于 mHC，扩展因子设为 4，Sinkhorn-Knopp 迭代次数设为 20。对于视觉编码器，我们将其层数设为 32，隐藏维度设为 1024，注意力头数设为 16，图像块大小设为 14。视觉 MLP 投影器有 2 层，隐藏维度为 5120。在此配置下，DeepSeek-V4.1-Flash 包含 552B 骨干参数，预填充阶段每个 token 激活 8B，解码阶段激活 16B。

We set the number of Transformer layers to 40 and the hidden dimension \(d\) to 5120. We adopt a Causal Encoder-Decoder architecture, with 20 layers in the encoder and 20 layers in the decoder. For the first two layers, we use pure sliding window attention. The remaining 18 encoder layers use CSA2 with a compression rate of \(m=2\). These layers are divided into three identically configured groups of six layers. In each group, the first layer operates in Full Mode, and the remaining five layers operate in Reuse Mode. The 20 decoder layers use CSA2 with a compression rate of \(m=1\). These layers are divided into five groups of four layers. In the first group, the first layer operates in Full Mode, and the remaining three layers operate in Reuse Mode. The remaining four groups share the same configuration: the first layer operates in Reindex Mode, and the remaining three layers operate in Reuse Mode. For all CSA2 layers, we set the number of indexer query heads to 32, the indexer head dimension to 128, and the number of KV entries selected for sparse attention (i.e., attention top-k) to 512. We set the number of query heads to 64, the head dimension to 512, and the query compression dimension to 1280. For the Hierarchical Sparse Indexer, we select a maximum of 2,048 blocks with 8 positions, yielding up to 16,384 candidate positions in total. The number of output projection groups is set to 8, and the dimension of each intermediate attention output is set to 1024. For the additional branch of sliding window attention, the window size \(n_{\mathrm{win}}\) is set to 128. We employ MoE layers in all Transformer blocks, using SwiGLU activation function with clamping at a threshold of 10. Each MoE layer consists of 1 shared expert and 384 routed experts, where the intermediate hidden dimension of each expert is 2304. Among the routed experts, 6 experts will be activated for each token. As for mHC, the expansion factor is set to 4, and the number of Sinkhorn-Knopp iterations is set to 20. For the vision encoder, we set its number of layers to 32, the hidden dimension to 1024, the number of attention heads to 16, and the image patch size to 14. The vision MLP projector has 2 layers with a hidden dimension of 5120. Under this configuration, DeepSeek-V4.1-Flash comprises 552B backbone parameters, with 8B activated per token during prefill and 16B during decode.

4.2.2 训练设置 4.2.2 Training Setups

我们对线性变换的参数采用 Muon 优化器，对所有 RMSNorm 模块的权重及其他非矩阵参数使用 AdamW 优化器，并对所有嵌入和预测头使用 Sinkhorn 平衡更新。对于 AdamW，我们将其超参数设置为 \(\beta_{1}=0.9\)、\(\beta_{2}=0.95\)、\(\varepsilon=10^{-20}\) 和 \(\mathrm{weight\_decay}=0.1\)。对于 Muon，我们将动量设置为 \(0.95\)，权重衰减设置为 \(0.1\)，并将每个更新矩阵的 RMS 重新缩放到 \(0.18\)，以复用 AdamW 的学习率。对于 Sinkhorn 平衡更新，我们使用与 Muon 相同的动量系数和学习率校正因子，并设置 \(K=11\)、\(\tau=10^{-3}\)、\(\varepsilon=10^{-20}\)。遵循 ，Engram 的学习率被缩放为 \(5\times\)。我们在 45T 多模态数据 token 上训练 DeepSeek-V4.1-Flash，未出现不稳定性。整个训练过程中，我们将批量大小固定为 1.006 亿 token。学习率在前 2000 步线性预热，然后保持为 \(2.6\times 10^{-4}\) 直到 28T token。在 28T 到 40T token 之间，我们按照余弦调度将学习率衰减到 \(2.6\times 10^{-5}\)。从 40T 到 45T token，我们将学习率保持在该值。我们从头开始训练模型，使用稀疏注意力机制，序列长度为 64K，并在 34T token 时将序列长度扩展到 1M。对于无辅助损失的负载均衡，我们将图像和文本 token 的偏置更新速度设置为 0.001，同时保留一个小的序列级平衡损失，损失权重为 0.0001，以避免单个序列内的极端不平衡。与 DeepSeek-V4 类似，我们在预训练期间采用样本级注意力掩码。

We employ the Muon optimizer for the parameters of linear transformations, use the AdamW optimizer for the weights of all RMSNorm modules and other non-matrix parameters, and use Sinkhorn-balanced update for all embeddings and prediction head. For AdamW, we set its hyper-parameters to \(\beta_{1}=0.9\), \(\beta_{2}=0.95\), \(\varepsilon=10^{-20}\), and \(\mathrm{weight\_decay}=0.1\). For Muon, we set the momentum to \(0.95\) and the weight decay to \(0.1\), and rescale the RMS of each update matrix to \(0.18\) for reutilization of the AdamW learning rate. For Sinkhorn-balanced updating, we use the same momentum coefficient and learning-rate correction factor as for Muon and set \(K=11\), \(\tau=10^{-3}\), \(\varepsilon=10^{-20}\). Following , the learning rate of Engram is scaled by \(5\times\). We train DeepSeek-V4.1-Flash on 45T tokens of multimodal data with no instability. We keep the batch size fixed at 100.6 million tokens throughout training. The learning rate is linearly warmed up over the first 2000 steps and then maintained at \(2.6\times 10^{-4}\) until 28T tokens. Between 28T and 40T tokens, we decay the learning rate to \(2.6\times 10^{-5}\) following a cosine schedule. We keep the learning rate at this value from 40T to 45T tokens. We train the model from scratch with sparse attention at a sequence length of 64K and extend the sequence length to 1M at 34T tokens. For auxiliary-loss-free load balancing, we set the bias update speed to 0.001 for both image and text tokens, while retaining a small sequence-level balance loss with a loss weight of 0.0001 to avoid extreme imbalance within single sequences. Similar to DeepSeek-V4, we employ sample-level attention masking during pre-training.

我们的 DeepSeek-ViT 在与语言主干网络集成之前，会经历一个单独的训练阶段。训练流程包括两个阶段：对比预训练和自回归微调。在对比预训练期间，我们使用 SigLIP 引入的 sigmoid 对比损失，在来自 alt-text 数据的大约 47B 图像-文本对上优化模型。为了从如此大规模的数据集中高效学习视觉表示，我们通过缩小较大图像同时保持其宽高比，将最大输入分辨率限制为 \(224\times 224\) 像素。尽管在此阶段使用更高分辨率能带来显著增益，但实证结果表明这些收益对最终模型贡献甚微。由于后续的自回归阶段专门处理高分辨率外推，在对比预训练期间提高分辨率会显著增加计算开销，而整体改进不大。在自回归微调阶段，我们将视觉编码器连接到 4B MoE LLM，并在包括图像描述、alt text、图表和 OCR 在内的数据集上，使用下一个词预测目标训练 236B token。此阶段旨在增强编码器建模细粒度视觉特征的能力。因此，我们通过按比例缩放超出边界的图像，将输入分辨率限制在 \(544\times 544\) 到 \(1344\times 1344\) 像素之间。在此阶段之后，我们丢弃 LLM，仅保留优化后的视觉编码器用于后续预训练流程，其中保持相同的输入分辨率策略。

Our DeepSeek-ViT undergoes a separate training stage before being integrated with the language backbone. The training pipeline consists of two stages: contrastive pre-training and autoregressive fine-tuning. During contrastive pre-training, we optimize the model using the sigmoid contrastive loss introduced by SigLIP on approximately 47B image-text pairs sourced from alt-text data. To efficiently learn visual representations from such massive datasets, we restrict the maximum input resolution to \(224\times 224\) pixels by downscaling larger images while preserving their aspect ratios. Although using higher resolutions in this phase yields notable gains, empirical results show that these benefits contribute little to the final model. Because the subsequent autoregressive stage specifically handles high-resolution extrapolation, scaling up resolutions during contrastive pre-training significantly increases computational overhead without much overall improvement. In the autoregressive fine-tuning stage, we connect the vision encoder to a 4B MoE LLM and train on 236B tokens across datasets including image captions, alt text, charts, and OCR, using a next-token prediction objective. This stage aims to enhance the encoder’s ability to model fine-grained visual features. We therefore constrain the input resolution between \(544\times 544\) and \(1344\times 1344\) pixels by proportionally scaling out-of-bound images. After this stage, we discard the LLM and retain only the optimized vision encoder for the subsequent pre-training pipeline, where the same input-resolution policy is maintained.

4.3.1 评估基准 4.3.1 Evaluation Benchmarks

我们将 DeepSeek-V4.1-Flash-Base 与其前代模型 DeepSeek-V4-Flash-Base 和 DeepSeek-V4-Pro-Base 进行比较。我们报告了涵盖五个关键维度的基准测试：世界知识、语言理解与推理、编程与数学、长上下文以及多模态能力。

We compare DeepSeek-V4.1-Flash-Base with its predecessor models DeepSeek-V4-Flash-Base and DeepSeek-V4-Pro-Base. We report benchmarks spanning five key dimensions: world knowledge, language understanding and reasoning, coding and mathematics, long context, and multimodal abilities.

世界知识基准包括 AGIEval、MMLU-Pro、C-Eval、MultiLoKo、SimpleQA-Verified 和 SuperGPQA。

World knowledge benchmarks include AGIEval, MMLU-Pro, C-Eval, MultiLoKo, SimpleQA-Verified, and SuperGPQA.

语言理解与推理基准包括 BigBench Hard (BBH)、BigBench Extra Hard (BBEH)、DROP 和 HellaSwag。

Language understanding and reasoning benchmarks include BigBench Hard (BBH), BigBench Extra Hard (BBEH), DROP, and HellaSwag.

编程与数学基准包括 BigCodeBench、HumanEval、GSM8K、MATH 和 MGSM。

Coding and mathematical benchmarks include BigCodeBench, HumanEval, GSM8K, MATH, and MGSM.

长上下文基准包括 LongBench-V2。

Long context benchmark includes LongBench-V2.

多模态基准包括 MMMU-Pro、DocVQA、CVBench 以及 RefCOCO/RefCOCO+/RefCOCO-g。

Multimodal benchmarks include MMMU-Pro, DocVQA, CVBench, and RefCOCO/RefCOCO+/RefCOCO-g.

4.3.2 评估结果 4.3.2 Evaluation Results

在表 1 中，我们详细比较了 DeepSeek-V4-Flash、DeepSeek-V4-Pro 和 DeepSeek-V4.1-Flash 的基础模型，所有模型均在我们内部的评估框架下，使用严格控制和可复现的设置进行评估。与 DeepSeek-V4-Flash-Base 和 DeepSeek-V4-Pro-Base 相比，我们最新的基础模型展现出显著的效率提升。DeepSeek-V4.1-Flash 激活的参数数量远少于 DeepSeek-V4-Pro-Base，且占用的 KV 缓存大幅减少，但其性能完全与前代模型持平。这些结果也反映了我们在预训练数据整理流程方面所做的重大改进。在此版本中，我们引入了原生多模态训练，并通过相应的多模态评估验证了其有效性。在更多样化和多模态的语料库上训练后，DeepSeek-V4.1-Flash 达到了与 DeepSeek-V4-Pro 相当的世界知识和理解能力。在推理和编码基准测试中，DeepSeek-V4.1-Flash 表现出持续的进步，在多个基准测试上达到接近或优于 DeepSeek-V4-Pro 的性能。

In Table 1, we provide a detailed comparison of the base models for DeepSeek-V4-Flash, DeepSeek-V4-Pro and DeepSeek-V4.1-Flash, all evaluated under our internal evaluation framework using strictly controlled and reproducible settings. Compared with DeepSeek-V4-Flash-Base and DeepSeek-V4-Pro-Base, our latest base model reveals a compelling efficiency gain. DeepSeek-V4.1-Flash activates a substantially smaller number of parameters than DeepSeek-V4-Pro-Base and occupies a heavily-reduced KV cache, yet its performance is fully on par with its predecessors. These results also reflect the substantial improvements we made to our pre-training data curation pipeline. In this version, we introduce native multimodal training and validate its effectiveness through corresponding multimodal evaluations. Trained on a more diverse and multimodal corpus, DeepSeek-V4.1-Flash achieves world knowledge and comprehension capabilities comparable to those of DeepSeek-V4-Pro. In reasoning and coding benchmarks, DeepSeek-V4.1-Flash shows consistent progress, reaching performance close to or better than DeepSeek-V4-Pro across multiple benchmarks.

为了进一步评估模型在真实研发场景中的能力，我们还对专门的内部语料库进行了困惑度测试。由于通过模型 API 无法进行困惑度测试，我们主要关注自己预训练的基础模型。在语料选择上，我们基于日常开发收集了一个单独的评估集，包括内部文档、专有代码库和学术材料，针对复杂科学问题和前沿研究的推理、归因和问题解决能力。结果如图 6 所示，我们报告了不同模型的每字节比特数（BPB），数值越低表示性能越好。

To further assess the model’s capabilities in real-world R&D scenarios, we additionally perform perplexity tests on dedicated internal corpora. Since perplexity tests are impossible through model APIs, we mainly focus on our own pretrained base models. For corpus selection, a separate evaluation set is collected based on our daily development, including internal documentation, proprietary code repositories and academic materials, which targets reasoning, attribution and problem-solving on complex scientific problems and frontier research. The results are shown in Figure 6, where we report the bits-per-byte (BPB) of different models, with lower values indicating better performance.

5.1 后训练流程 5.1 Post-Training Pipeline

在本版本中，我们避免引入新颖的后训练算法。整体方案遵循标准范式：先进行监督微调（SFT），随后进行强化学习（RL）和同策略蒸馏（OPD；），除成熟实践外未做算法修改。相反，我们的努力几乎完全集中在模型训练所用的数据上，而非优化方式：我们投入大规模自动化流程进行数据合成和环境构建。具体而言，该流程（i）合成多样、可验证的训练任务及其参考解和奖励信号，（ii）程序化构建并扩展交互式智能体环境，在其中可以低成本收集和评估轨迹，以及（iii）应用严格的过滤、去重和难度校准，以确保数据质量和课程平衡。我们发现，在固定且普通的优化过程下，合成数据和环境在规模、多样性和可验证性方面的系统性改进几乎解释了所有观察到的收益。这一观察呼应了一个更广泛的教训：在当前阶段，工程化数据和环境流程的边际回报显著超过后训练中算法新颖性的边际回报。

In this release, we refrain from introducing novel post-training algorithms. The overall recipe follows the standard paradigm of supervised fine-tuning (SFT) followed by reinforcement learning (RL) and on-policy distillation (OPD; ), without algorithmic modifications beyond well-established practices. Instead, our efforts are concentrated almost entirely on what the model is trained on rather than how it is optimized: we invest in large-scale, automated pipelines for data synthesis and environment construction. Concretely, the pipeline (i) synthesizes diverse, verifiable training tasks together with their reference solutions and reward signals, (ii) procedurally constructs and scales interactive agent environments in which trajectories can be collected and evaluated at low cost, and (iii) applies rigorous filtering, deduplication, and difficulty calibration to ensure data quality and curriculum balance. We find that, under a fixed and unremarkable optimization procedure, systematic improvements in the scale, diversity, and verifiability of synthesized data and environments account for essentially all of the observed gains. This observation echoes a broader lesson: at the current stage, the marginal return of engineering the data and environment pipeline substantially exceeds that of algorithmic novelty in post-training.

5.1.1 大规模智能体任务合成 5.1.1 Large-Scale Agent Task Synthesis

任务是智能体学习的根本燃料。然而，构建高质量的训练任务传统上需要大量人工投入。我们观察到，模型已经开始展现出构建自身训练任务的能力，尽管这一能力仍远未完善。认识到这一潜力后，我们投入了大量精力来增强模型的任务构建与质量验证能力。

Tasks serve as the fundamental fuel for agent learning. However, constructing high-quality training tasks has traditionally required substantial manual effort. We observe that the model is already beginning to exhibit the ability to construct its own training tasks, though this capability remains far from perfect. Recognizing this potential, we have invested considerable effort in strengthening the model's task-construction and quality-verification abilities.

我们将每个任务形式化为一个三元组（问题、环境、验证系统），并从两个维度评估其质量：难度——确保任务并非平凡——以及正确性——保证三个组成部分中不存在关键缺陷。以难度和正确性作为奖励信号，我们迭代训练模型以构建更好的任务。我们还对强化学习任务进行全生命周期监控。每当一个任务被用于新的强化学习运行时，所产生的轨迹为质量复审提供了新的证据。在这一通用框架下，我们为两个核心场景构建了专门的训练环境生产流水线：通用智能体和编码智能体。

We formalize each task as a triplet (problem, environment, verification system) and evaluate its quality along two dimensions: difficulty—ensuring the task is non-trivial—and correctness—guaranteeing that no critical flaws exist among the three components. Using difficulty and correctness as reward signals, we iteratively train the model to construct better tasks. We also monitor RL tasks across their full lifecycle. Whenever a task is used in a new RL run, the resulting trajectories provide fresh evidence for quality re-auditing. Under this general framework, we have built dedicated training environment production pipelines for two core scenarios: general agents and coding agents.

通用智能体。对于通用智能体，我们鼓励内部员工和外部合作伙伴将我们的最新模型融入其日常工作流程，并在自愿基础上返回交互数据和反馈。基于返回数据中观察到的接口，我们构建了大量模拟工具，复现真实世界工具和系统的接口与行为，包括其输入格式、输出结构、API 模式和行为约束，覆盖常用的 SaaS 和企业应用以及更专业的业务后端系统。与此同时，我们大规模收集内部员工提交的负面反馈和模型失败案例，并将其纳入流水线，以生成基于真实工作流的单轮和多轮智能体环境。通过重建相关的工具上下文、用户交互模式和失败条件，该流水线能够系统性地重放失败，并针对观察到的模型弱点进行定向强化学习。

General Agent. For general agents, we encourage internal employees and external partners to incorporate our latest model into their routine workflows and, on a voluntary basis, return interaction data and feedback. Based on the interfaces observed in the returned data, we construct a large set of mocked tools that reproduce the interfaces and behaviors of real-world tools and systems, including their input formats, output structures, API schemas, and behavioral constraints, covering both commonly used SaaS and enterprise applications as well as more specialized business back-end systems. In parallel, we collect negative feedback and model failure cases submitted by internal employees at scale, and incorporate them into the pipeline to generate both single-turn and multi-turn agent environments grounded in real workflows. By reconstructing the relevant tool context, user interaction patterns, and failure conditions, the pipeline enables systematic replay of failures and targeted reinforcement learning against observed model weaknesses.

编码智能体。编码智能体训练环境来自两个来源：1）来自内部员工和外部合作伙伴的编码智能体会话，经过筛选保留高度复杂的任务或模型表现较差的任务，然后按轨迹去重；2）满足星标数量阈值的公开 GitHub 仓库。环境构建由多个专门智能体协作完成。首先，一个智能体判断项目是否可以在容器内构建并完整运行，以及是否可以自动验证；如果可以，它选择一个特定的轮次或提交作为任务起点，设计若干足够复杂的实现方向，并产出具体的评估点，包括 fail-to-pass 和 pass-to-pass 点，以及一份构建报告，并按需从网络获取外部资源。接下来，另一个智能体在隔离容器中设置依赖、初始工作目录、测试代码和任务描述，进行自测，移除任何可能泄露任务解决方案的痕迹，并将环境打包为新的镜像层。然后，多个不同的智能体尝试该任务，一个独立的质量检查智能体审查环境以及解题智能体的轨迹，检查环境问题、事实错误、评估点与任务描述之间的不匹配以及可作弊风险。如果检查未通过，一个修复智能体修复所有已识别的错误，调整过于简单或过于困难的评估点，然后任务重新进入验证。

Coding Agent. Coding agent training environments are built from two sources: 1) coding-agent sessions from internal employees and external partners, filtered to retain highly complex tasks or tasks on which model performance is poor, then deduplicated by trajectory; and 2) public GitHub repositories that meet a star-count threshold. Environment construction is carried out collaboratively by multiple specialized agents. First, an agent determines whether the project can be built and fully run inside a container and whether it can be automatically verified; if so, it selects a specific turn or commit as the task starting point, designs several sufficiently complex implementation directions, and produces concrete evaluation points, including both fail-to-pass and pass-to-pass points, along with a construction report, fetching external resources from the web as needed. Next, a separate agent sets up dependencies, the initial working directory, test code, and task descriptions in an isolated container, performs self-testing, removes any traces that could leak the task solution, and packages the environment as a new image layer. Then, multiple distinct agents attempt the task, and an independent quality-inspection agent reviews the environment together with the solving agents' trajectories, checking for environment issues, factual errors, mismatches between evaluation points and task descriptions, and hackability risks. If the inspection does not pass, a repair agent fixes all identified errors, adjusts evaluation points that are too easy or too difficult, and the task re-enters verification.

通过这些流水线，我们可以自动、批量地生产正确、有区分度且长度和难度可控的强化学习训练数据。无论是通过通用智能体环境中对真实工作流的忠实重建，还是编码智能体环境中对编码任务的精确构建，两者最终都汇入一个统一的训练系统，在持续的质量监控下推动模型能力的迭代提升。

Through these pipelines, we can automatically and batch-produce RL training data that is correct, discriminative, and controllable in length and difficulty. Whether through the faithful reconstruction of real workflows in general agent environments or the precise construction of coding tasks in coding agent environments, both ultimately feed into a unified training system, driving iterative improvement of model capabilities under continuous quality monitoring.

5.1.2 合成任务中的强化学习 5.1.2 RL in Synthesized Tasks

我们在合成任务中使用大规模异步强化学习，以提升模型性能并塑造其在复杂场景中的行为。我们从两个维度扩展强化学习运行，即训练算力和脚手架数量。如图 7 和图 8 所示，随着累积强化学习步数增加算力，性能持续提升，无论是在单个脚手架内扩展、在同一脚手架的变体间联合扩展，还是在异构脚手架间扩展。

We use large-scale asynchronous RL in synthesized tasks to improve model performance and shape its behavior in complicated scenarios. We scale RL runs in two dimensions, i.e., training compute and the number of scaffolds. As shown in Figure 7 and Figure 8, performance continues to improve as we increase compute with cumulative RL steps, whether scaling within a single scaffold, jointly across variants of the same scaffold, or across heterogeneous scaffolds.

为了在多样化脚手架间进行强化学习训练，我们将智能体 rollout 执行解耦为智能体沙箱和工作者容器。沙箱运行脚手架及其工具，而工作者提供一个与脚手架无关的控制层，负责编排 rollout、将异构交互规范化为统一的轨迹模式，并与训练器通信。两者都运行在 DSec（第 5.1.3 节）上，位于可抢占的 GPU 训练池之外，将长时 rollout 与细粒度训练调度分离。在训练器被抢占期间，rollout 执行可以暂停并卸载，同时保留其完整状态以便后续恢复，并释放 CPU 和 GPU 资源。这种设计使得在异构脚手架间进行稳定高效的强化学习成为可能，而无需修改底层算法。

For RL training across diverse scaffolds, we decouple agent rollout execution into an agent sandbox and a worker container. The sandbox runs the scaffold and its tools, while the worker provides a scaffold-agnostic control layer that orchestrates the rollout, normalizes heterogeneous interactions into a common trajectory schema, and communicates with the trainer. Both run on DSec (Section 5.1.3), outside the preemptible GPU training pool, separating long-lived rollouts from fine-grained training scheduling. During trainer preemption, rollout execution can be suspended and offloaded while preserving its full state for later resumption and releasing CPU and GPU resources. This design enables stable and efficient RL across heterogeneous scaffolds without modifying the underlying algorithms.

为了将有效的强化学习算力扩展到单次训练运行之外，我们使用模型合并来重新初始化后续的强化学习运行。具体来说，我们合并来自不同脚手架或配置的运行检查点，结合沿不同优化路径获得的改进。在图 7 和图 8 中，不连续的曲线段反映了模型重新初始化后的连续强化学习运行。这在任务性能和 token 效率方面都带来了进一步收益，提供了一种简单实用的方法来聚合并行强化学习算力，并在连续运行中持续扩展。

To extend effective RL compute beyond a single training run, we use model merging to reinitialize successive RL runs. Specifically, we merge checkpoints from runs across different scaffolds or configurations, combining improvements acquired along different optimization paths. In Figure 7 and Figure 8, disconnected curve segments reflect successive RL runs after model reinitialization. This yields further gains in both task performance and token efficiency, providing a simple and practical way to aggregate parallel RL compute and continue scaling across successive runs.

5.1.3 大规模运行智能体：DSec 5.1.3 Running Agents at Massive Scale: DSec

随着我们从 DeepSeek-V3 过渡到 V4，智能体训练环境的数量和多样性迅速增长，促使我们构建了 DeepSeek Elastic Compute（DSec），这是一个面向大规模智能体训练与评估的生产级沙箱平台。其初始设计解决了异构执行环境、可扩展镜像分发、多种隔离后端、高密度资源管理、命令轨迹日志记录以及抢占安全恢复等问题。

As we transitioned from DeepSeek-V3 to V4, the rapidly growing number and diversity of agentic training environments motivated us to build DeepSeek Elastic Compute (DSec), a production-grade sandbox platform for large-scale agentic training and evaluation. Its initial design addressed heterogeneous execution environments, scalable image distribution, multiple isolation backends, high-density resource management, command trajectory logging, and preemption-safe resumption.

V4.1 的训练进一步将需求提升至数百万个并发沙箱实例，涵盖多样化的执行框架、平台、代码仓库、软件依赖和任务特定服务。在此规模下，主要瓶颈转向数据中心可扩展性、工作负载隔离、单节点算力密度，以及对能力日益增强的智能体不当行为的遏制。我们简要描述关键设计要点如下。

V4.1 training further increased demand to millions of concurrent sandbox instances spanning diverse harnesses, platforms, code repositories, software dependencies, and task-specific services. At this scale, the primary bottlenecks shifted toward datacenter scalability, workload isolation, per-node compute density, and the containment of misbehavior by increasingly capable agents. We briefly describe our key design aspects below.

大规模水平扩展算力。DSec 通过两种互补机制实现扩展：分片和松弛一致性调度。为了容纳大量机器，我们将计算节点划分为多个分片（即所谓的规模单元）。这种分片还允许我们通过隔离不同实验的工作负载来减少爆炸半径，防止单个内存密集型任务耗尽无关工作共享的资源。

Horizontally Scaling Compute at Scale. DSec scales through two complementary mechanisms: sharding and scheduling with relaxed consistency. To accommodate a large number of machines, we partition compute nodes into multiple shards (so-called scale units). Such sharding also allows us to reduce blast radius by isolating workloads from different experiments, preventing a single memory-intensive task from exhausting resources shared by unrelated work.

DSec 没有使用像 Kubernetes 这样的现成编排器，而是采用自定义放置引擎来调度大量沙箱，通过牺牲强全局一致性来换取可扩展性。这一设计基于一个关键观察：只要每个计算节点强制执行本地安全约束，智能体沙箱放置仅需最终一致性。为此，放置引擎部署多个独立副本，无需同步协调。每个副本根据近期测量预测资源可用性，并做出足够好的放置决策。为了补偿一致性的损失，每个节点负责验证最终放置决策，强制执行硬性准入约束，如果超过本地警告阈值则拒绝新放置。整体设计使 DSec 能够扩展到数百万个容器，而不会在中央协调上出现瓶颈。

Instead of using off-the-shelf orchestrators like Kubernetes, DSec employs a custom placement engine to schedule the large number of sandboxes, by trading strong global consistency for scalability. This design is based on a key observation: agentic sandbox placements only demand eventual consistency as long as each compute node enforces local safety constraints. To do so, the placement engine deploys in multiple independent replicas, without synchronized coordination. Each replica predicts resource availability from recent measurements and makes good-enough placement decisions. To compensate for this loss in consistency, each node is responsible for validating the final placement decisions, enforcing a hard admission constraint that rejects new placements if it exceeds a local warning threshold. This design as a whole allows DSec to scale to millions of containers without bottlenecking on central coordination.

高密度运行沙箱。在节点层面，我们使用硬件支持的子 NUMA 分区，并将每个工作虚拟机绑定到单独的 NUMA 域。容器在这些工作虚拟机内运行，其 CPU 和内存分配限制在虚拟机的本地 NUMA 资源内。这种设置平衡了激进的内存超配与 Linux 内核锁争用，同时将内存压力和运行时故障局部化。在可比的工作负载配置下，它将每个物理节点支持的密度从约 1,000 个并发活动容器提高到超过 2,500 个，之后才会出现可测量的端到端性能下降。

Running Sandboxes at High Density. At the node level, we use hardware-supported sub-NUMA partitioning and bind each worker VM to an individual NUMA domain. Containers run within these worker VMs, with their CPU and memory allocations confined to the VM’s local NUMA resources. This setup balances aggressive memory overcommitment against Linux kernel lock contention, while localizing memory pressure and runtime failures. Under comparable workload configurations, it increases the supported density from roughly 1,000 to more than 2,500 concurrent live containers per physical node before measurable end-to-end degradation appears.

然而，这种高密度部署可能因后台工作负载的干扰而扭曲对时间敏感的评估。因此，DSec 引入了延迟敏感（LS）执行类。我们对非 LS 任务应用 SCHED_IDLE 以最小化其调度优先级，并使用核心调度确保只有相同优先级类的任务在兄弟超线程上同时执行，以消除干扰。

Such high-density deployment can nevertheless distort time-sensitive evaluations through interference from background workloads. DSec therefore introduces a latency-sensitive (LS) execution class. We apply SCHED_IDLE to non-LS tasks to minimize their scheduling priority, and use core scheduling to ensure only tasks of the same priority class execute simultaneously on sibling hyperthreads to eliminate interference.

**对行为不当智能体的缓解。** 在强化学习训练期间，我们经常观察到智能体试图进行奖励黑客行为或无意中使环境崩溃。在某些尝试中，我们的智能体利用了最近披露的漏洞，包括 XFS 驱动的权限问题、AppArmor 中的非法内存访问、从软件包镜像服务泄露答案等。智能体还以删除关键二进制文件、破坏系统文件甚至移除文件系统而臭名昭著。我们使用每个沙箱的 AppArmor 配置文件和基于 eBPF 的细粒度网络策略来防止此类尝试。如果智能体使其环境崩溃，我们将崩溃视为失败的轨迹，并向强化学习框架报告一个“反冲”信号。

Mitigation of Misbehaving Agents. During RL training, we frequently observe agents attempting to perform reward hacking or unintentionally crashing the environment. In certain attempts, our agents exploited recently disclosed vulnerabilities, including permission issues from the XFS driver, illegal memory access in AppArmor, leaking answers from package mirror services, and so on. Agents have also been notorious for deleting critical binaries, breaking system files, or even removing the filesystem. We use per-sandbox AppArmor profiles and fine-grained eBPF-based network policies to prevent such attempts. If an agent crashes its environment, we treat the crash as a failed trajectory and report a "repercussion" signal to the RL framework.

5.1.4 强化学习中的可控推理努力 5.1.4 Controllable Reasoning Effort in RL

除了模型架构和硬件的进步之外，输出 token 的数量是服务成本的另一个关键决定因素，因此也是实际应用中成本-质量权衡的关键因素。因此，我们在强化学习过程中引入一个标量努力水平 \(b\) 作为显式条件信号。该机制适用于单轮推理和多轮智能体式任务。具体来说，我们在系统提示前添加以下指令：

Alongside advances in model architecture and hardware, the number of output tokens is another key determinant of serving cost and, consequently, of the cost–quality trade-off in real-world applications. We therefore introduce a scalar effort level \(b\) as an explicit conditioning signal during reinforcement learning. This mechanism is applied to both single-turn reasoning and multi-turn agentic tasks. Specifically, we prepend the following instruction to the system prompt:

推理努力：{effort}（范围 1--100；值越高要求越彻底的推理）

Reasoning Effort: {effort} (range 1--100; higher values request more thorough reasoning)

这里，\(b \in \{1, \ldots, 100\}\) 表示请求的努力水平。

Here, \(b \in \{1, \ldots, 100\}\) denotes the requested effort level.

对于每个训练提示 \(x\)，我们在每个努力水平 \(b \in \mathcal{B}\) 上采样 \(M_b\) 个响应：

For each training prompt \(x\), we sample \(M_b\) responses at each effort level \(b \in \mathcal{B}\):

这里，\(j\) 索引在努力水平 \(b\) 上采样的响应。共享相同 \((x, b)\) 的响应形成一个子组，在该子组内奖励被均值中心化以计算组相对优势。因此，来自不同努力水平的响应不会直接比较。相反，通过使奖励的长度分量依赖于 \(b\)，在每个子组内诱导出依赖于努力的行为。具体来说，我们将长度惩罚项 \(r_{b,j}^{\mathrm{len}}\) 添加到响应 \(z_{b,j}\) 的奖励中：

Here, \(j\) indexes the responses sampled at effort level \(b\). Responses sharing the same \((x, b)\) form a subgroup, within which rewards are mean-centered to compute group-relative advantages. Thus, responses from different effort levels are not directly compared. Instead, effort-dependent behavior is induced within each subgroup by making the length component of the reward depend on \(b\). Specifically, we add the length-penalty term \(r_{b,j}^{\mathrm{len}}\) to the reward of response \(z_{b,j}\):

其中 \(\ell_{b,j}\) 是推理 token 的数量，\(L_{\mathrm{norm}}\) 是参考长度，\(C_{\max}\) 是对轨迹施加的最大扣减的上限。token 惩罚系数随请求的努力程度呈指数下降：

where \(\ell_{b,j}\) is the number of reasoning tokens, \(L_{\mathrm{norm}}\) is a reference length, and \(C_{\max}\) caps the maximum deduction applied to a trajectory. The token-penalty coefficient decreases exponentially with the requested effort:

其中 \(k_{0}\) 是不同努力水平下的基本惩罚系数，\(b_{min}\) 是 \(\mathcal{B}\) 的最小值，\(\overline{\Delta b}\) 是训练努力水平之间的平均间距，\(\lambda\) 控制惩罚衰减的速率。将 \(b\) 增加 \(\tau\) 会使惩罚系数乘以 \(e^{-1}\)。参数 \(k_{0}\) 控制朝向更短推理的整体压力，而较小的 \(\tau\) 会使惩罚衰减更快，并倾向于在不同努力水平之间产生更大的行为分离。附录 C 为 \(k(b)\) 的指数参数化提供了边际效用的动机。

where \(k_{0}\) is the basic penalty coefficient at different effort levels, \(b_{min}\) is the minimum value of \(\mathcal{B}\), \(\overline{\Delta b}\) is the average spacing between training effort levels, and \(\lambda\) controls the rate of penalty decay. Increasing \(b\) by \(\tau\) multiplies the penalty coefficient by \(e^{-1}\). The parameter \(k_{0}\) controls the overall pressure toward shorter reasoning, whereas a smaller \(\tau\) causes the penalty to decay more rapidly and tends to produce greater behavioral separation between effort levels. Appendix C provides a marginal-utility motivation for the exponential parameterization of \(k(b)\).

在部署时，标量 \(b\) 为模型的推理强度提供了一个灵活的控制接口。通过改变 \(b\)，单个模型检查点可以在学习到的成本–质量前沿上在不同运行区间之间移动，以适应不同的延迟、token 预算和解决方案质量要求。尽管训练仅使用有限的一组努力水平，但在部署时可以使用中间值来引出插值推理行为，从而为测试时资源分配提供一种细粒度且高效的机制。

At deployment time, the scalar \(b\) provides a flexible control interface over the model’s reasoning strength. By varying \(b\), a single model checkpoint can move between different operating regimes along the learned cost–quality frontier, adapting to different latency, token-budget, and solution-quality requirements. Although training uses only a finite set of effort levels, intermediate values can be used at deployment to elicit interpolated reasoning behaviors, providing a fine-grained and efficient mechanism for test-time resource allocation.

在我们于 2026 年 9 月启动的生产部署中，公共 API 暴露了三个预设的推理努力层级——max、high 和 low——它们直接映射到这个标量接口上。如表 2 所总结，这三个层级分别对应努力值 \(b=100\)、\(b=75\) 和 \(b=50\)，因此 API 用户可以在不改变模型权重或解码配置的情况下，在学习到的成本–质量前沿上选择一个运行点。

In our production deployment launched in September 2026, the public API exposes three preset reasoning-effort tiers—max, high, and low—which map directly onto this scalar interface. As summarized in Table 2, the three tiers correspond to effort values of \(b=100\), \(b=75\), and \(b=50\), respectively, so that API users select an operating point on the learned cost–quality frontier without any change to the model weights or decoding configuration.

5.2 异步后训练基础设施 5.2 Asynchronous Post-training Infrastructure

在 LLM 的强化学习 rollout 阶段，长尾问题一直是训练效率的主要瓶颈。为解决这一问题，我们扩展了后训练基础设施，以支持样本的异步生成；通过维持足够高的并发水平，这显著缓解了 rollout 阶段的长尾问题。目前，我们几乎所有的强化学习与 OPD 任务都已启用异步训练，rollout 效率也得到了大幅提升。

The long-tail problem during the rollout phase of RL for LLMs has consistently been a major bottleneck for training efficiency. To address this, we extend our post-training infrastructure to allow asynchronous generation of samples, which significantly mitigates the long-tail issue in the rollout phase by maintaining a sufficiently high level of concurrency. Asynchronous training is now enabled for nearly all our RL and OPD tasks, and rollout efficiency has improved substantially.

5.2.1 整体工作流程 5.2.1 Overall Workflow

我们将 rollout 和训练共置于相同的物理设备上，并分时执行，从而无需手动调整两个阶段之间的资源分配。每个任务指定了在途样本数量的上限，系统在整个 rollout 阶段维持该上限。

We colocate rollout and training on the same physical devices and time-share their execution, eliminating the need to manually tune resource allocation between the two phases. Each task specifies an upper bound on the number of in-flight samples, and the system maintains this bound throughout the rollout phase.

我们评估了三种调度粒度以维持目标 rollout 并发数。最终采用的方法是样本级调度：一旦新完成的样本数量达到分配给下一个 prompt 的 GRPO 组大小，我们就调度该 prompt，无论这些完成样本来自哪些组。这有助于在整个训练过程中保持稳定的 rollout 并发数。在确定此调度策略之前，我们尝试了两种替代的调度粒度。第一次尝试中，我们在开始时调度了多个额外批次，并在每次训练迭代后补充一个完整批次；然而，这导致训练指标剧烈振荡，表明批次级粒度太粗。第二次尝试中，我们改用 prompt 级调度，即在一个 GRPO 组完成后调度一个新的 prompt，但发现它容易在 GRPO 组内的长尾样本上停滞，难以平稳维持目标 rollout 并发数。

We evaluated three dispatch granularities for maintaining the target rollout concurrency. Our final approach is sample-level dispatch: once the number of newly completed samples reaches the GRPO group size assigned to the next prompt, we dispatch that prompt regardless of which groups produced those completions. This helps maintain a steady rollout concurrency throughout training. Before settling on this dispatch strategy, we experimented with two alternative dispatch granularities. In the first attempt, we dispatched several extra batches at the beginning and supplemented a full batch after each training iteration; however, this caused severe oscillations in training metrics, indicating that batch-level granularity was too coarse. In the second attempt, we switched to prompt-level dispatch, where a new prompt was dispatched after one GRPO group finishes, but found that it stalled easily on long-tail samples within a GRPO group, making it difficult to smoothly maintain the target rollout concurrency.

一旦积累了足够的训练样本，训练会抢占正在进行的 rollout。在训练期间，我们使用拼接式路由重放：对于跨越多个检查点的样本，我们拼接每个 rollout 段产生的专家路由，而不是丢弃并用新检查点重新计算路由信息。

Once enough training samples have accumulated, training preempts ongoing rollouts. During training, we use concatenated routing-replay: for samples that span multiple checkpoints, we concatenate the expert routing produced at each rollout segment rather than discarding and recomputing the routing information with new checkpoints.

5.2.2 缓解长度偏差与离策略效应 5.2.2 Mitigating Length Bias and Off-Policy Effects

异步生成虽然有效提高了 rollout 效率，但引入了两个可能降低训练质量的副作用。首先，它会造成长度分布偏差，尤其是在训练早期阶段，因为较短的序列往往先完成，从而主导了初始训练批次。其次，它不可避免地产生离策略样本，即其 token 部分或全部由较早的检查点生成的样本。这两个问题需要不同的处理策略。

Asynchronous generation, while effectively improving rollout efficiency, introduces two side effects that can degrade training quality. First, it creates a length-distribution bias, especially in the early training stage, because shorter sequences tend to complete first and thus dominate the initial training batches. Second, it inevitably produces off-policy samples, i.e., samples whose tokens are partially or entirely generated by earlier checkpoints. These two issues require different handling strategies.

为了解决长度偏差，我们采用了两种机制。首先，调度器可以基于每个数据集限制并发数，这有助于调节稳态训练批次中每个数据集的比例，通过控制传入样本的来源间接缓解长度偏斜。其次，我们支持丢弃提前返回的短样本，以平滑过渡到稳态长度分布，并防止模型对过短序列过拟合。

To address the length bias, we employ two mechanisms. First, the dispatcher can limit concurrency on a per-dataset basis, which helps regulate the proportion of each dataset in the steady-state training batch, indirectly mitigating the length skew by controlling the sources of incoming samples. Second, we support discarding early-returned short samples to smooth the transition into the steady-state length distribution, and prevent the model from overfitting to overly short sequences.

对于离策略问题，我们实现了另外两种机制。首先，通过调整控制样本调度和训练样本等待条件的逻辑，我们可以限制最大离策略比例，确保训练数据不会过度偏离当前模型。其次，在训练过程中，我们添加了一种损失掩码方案，消除过度陈旧 token 的贡献，从而减轻陈旧样本对梯度更新的不利影响。

For the off-policy issue, we implement two additional mechanisms. First, by tuning the logic that controls sample dispatching and the waiting condition for training samples, we can bound the maximum off-policy ratio, ensuring that the training data does not deviate excessively from the current model. Second, during training, we add a loss masking scheme that eliminates the contribution of tokens with excessive staleness, thereby mitigating the adverse impact of stale samples on gradient updates.

5.2.3 性能优化 5.2.3 Performance Optimization

在我们的异步强化学习框架中，rollout 阶段会定期中断，以切换到更新后的策略检查点。我们力求使这一过程无缝衔接：中断应近乎瞬时完成，且被中断的 rollout 应能像从未停止过一样恢复。

In our asynchronous RL framework, the rollout phase is periodically interrupted to switch to updated policy checkpoints. We aim to make this process seamless: interruptions should be near-instantaneous, and interrupted rollouts should resume as if never stopped.

为了及时停止 rollout，我们支持 token 级中断：生成过程可以在任意 token 边界处暂停。一旦收集到足够的训练数据，所有进行中的样本几乎立即停止，使系统能够无延迟地进入训练阶段。

To stop rollouts promptly, we support token-level interruption: generation can be halted at any token boundary. Once sufficient training data has been collected, all in-flight samples stop almost immediately, allowing the system to enter the training phase without delay.

为了在检查点切换期间保留 rollout 进度，生成过程中会以 token 粒度持久化 rollout 状态，如 KV 缓存和专家路由。使用新检查点恢复时，直接复用持久化的状态，从而消除重新预填充的开销，并允许被中断的样本从上次中断处精确继续。由于这需要保留所有进行中样本的状态，我们执行样本粒度的垃圾回收，一旦某个样本完成就立即释放其状态。

To preserve rollout progress across checkpoint switches, rollout states such as KV cache and expert routing are persisted at token granularity during generation. Upon resumption with a new checkpoint, the persisted states are directly reused, eliminating the cost of re-prefilling and allowing interrupted samples to continue exactly where they left off. Since this requires retaining the states of all in-flight samples, we perform sample-grained garbage collection, releasing each sample's states as soon as it completes.

除了检查点切换之外，相同的快速中断和无缝恢复机制还使训练任务能够及时响应集群调度抢占信号而不会丢失进度，从而提高整体集群利用率。

Beyond checkpoint switching, the same fast-interruption and seamless-resumption machinery allows the training jobs to respond promptly to cluster scheduling preemption signals without losing progress, thereby improving overall cluster utilization.

5.2.4 大规模同策略蒸馏 5.2.4 Large-Scale On-Policy Distillation

作为后训练的最后一个阶段，最终的全词表 OPD 任务使用来自所有领域的数据集，并借助超过 40 个教师模型进行训练。该阶段还采用异步生成以提高 rollout 效率。由于各领域的训练流程存在差异，每个领域的最佳教师可能来自模型开发的不同阶段。此外，教师模型之间以及教师与学生模型之间在架构上可能有所不同。我们的后训练基础设施能够轻松适应这种设置，支持全词表 OPD 使用数量实际上不受限制、架构异构的教师模型，并以可忽略的成本在它们之间高效切换。

As the last stage of post-training, the final full-vocabulary OPD task is trained on datasets from all domains using over 40 teacher models. It also adopts asynchronous generation to improve rollout efficiency. Due to differences in training procedures across domains, the best teacher for each domain may come from a different stage of model development. Moreover, the teacher models may differ architecturally from one another and from the student. Our post-training infrastructure readily accommodates this setting, supporting full-vocabulary OPD with an effectively unbounded number of architecturally heterogeneous teachers, and efficient switching among them at negligible cost.

OPD 阶段还需要在训练过程中进行动态重配置。我们持续跟踪模型能力，并可能相应地调整训练配方，包括数据集混合、每个数据集的并发限制以及活跃教师。在同步训练中，此类更改很简单，因为 rollout 批次提供了明确的配置边界。然而，在异步设置中，不同配置下生成的样本可能同时在途。我们的基础设施支持配置之间的一致转换，而不会中断 rollout 或训练。

The OPD stage also requires dynamic reconfiguration during training. We continuously track model capabilities and may adjust the training recipe accordingly, including the dataset mixture, per-dataset concurrency limits, and active teachers. Such changes are straightforward in synchronous training where rollout batches provide explicit configuration boundaries. In the asynchronous setting, however, samples generated under different configurations may coexist in flight. Our infrastructure supports consistent transitions between configurations without disrupting rollout or training.

5.3.1 评估设置 5.3.1 Evaluation Setup

我们的后训练评估主要聚焦于推理和智能体式能力，而知识密集型性能主要由预训练决定，并在表 1 中报告。对于推理，我们在 GPQA Diamond、Humanity's Last Exam、Codeforces（内部基准）和 MathArena Apex 上进行评估，使用温度参数和 top-\(p\)为 1.0。对于智能体式能力，我们评估四个类别：

Our post-training evaluation focuses primarily on reasoning and agentic capabilities, while knowledge-intensive performance is largely determined by pretraining and reported in Table 1. For reasoning, we evaluate on GPQA Diamond, Humanity's Last Exam, Codeforces (internal benchmark), and MathArena Apex, using temperature and top-\(p\) of 1.0. For agentic capabilities, we evaluate across four categories:

代码智能体：Terminal-Bench 2.1、Terminal-Bench 3.0、Terminal-Bench 4.0、DeepSWE v1.1、ProgramBench、NL2Repo-Bench。

Code agent: Terminal-Bench 2.1, Terminal-Bench 3.0, Terminal-Bench 4.0, DeepSWE v1.1, ProgramBench, NL2Repo-Bench.

网络安全：SEC-Bench Pro 版本 260505、CyberGym 和 ExploitGym。

Cyber security: SEC-Bench Pro version 260505, CyberGym, and ExploitGym.

通用智能体：AutomationBench v1.0.6 的公开评估集、Agents' Last Exam (ALE-CLI)。

General agent: the public evaluation set of AutomationBench v1.0.6, Agents' Last Exam (ALE-CLI).

视觉智能体：Chartography、BabyVision、ZeroBench 主集。

Visual agent: Chartography, BabyVision, main set of ZeroBench.

对于代码智能体，我们使用 DeepSeek Harness 的 Minimal 模式评估 DeepSeek-V4.1-Flash，上下文窗口为 1M token，温度设为 1.0，top-p 设为 0.95。为符合官方设置要求，我们对 DeepSWE v1.1 采用 mini-SWE harness。对于 SEC-Bench Pro，我们专门利用 Claude Code harness 的会话压缩设计。对于视觉智能体任务，我们使用 Claude Code harness 进行评估，上下文窗口为 512k token，温度设为 1.0，top-p 设为 0.95。Agents' Last Exam 和 AutomationBench 使用其官方脚手架进行评估。使用其他编码脚手架时的模型性能报告在表 4 中。

For code agents, we evaluate DeepSeek-V4.1-Flash using the Minimal mode of DeepSeek Harness with a 1M-token context window, temperature set to 1.0, and top-p set to 0.95. To align with official setup requirements, we employ the mini-SWE harness for DeepSWE v1.1. For SEC-Bench Pro, we utilize the Claude Code harness specifically for its session compact design. For visual agent tasks, we evaluate using the Claude Code harness with a 512k-token context window, temperature set to 1.0, and top-p set to 0.95. Agents' Last Exam and AutomationBench are evaluated with their official scaffolds. Model performance with other coding scaffolds is reported in Table 4.

为减轻代码智能体评估中的奖励黑客行为，我们限制互联网访问并从环境中剥离 Git 历史。此外，我们自动清除跨不同环境的临时构建和包缓存，包括 Go 模块缓存（go/mod）、node modules 依赖产物、编译的 .jar 文件以及 Python pycache 目录。尽管采取了这些预防措施，我们在测试中仍观察到寻求漏洞利用的行为实例——例如在 CyberGym 中反编译 Ubuntu Linux 核心包以发现漏洞。随着模型能力日益增强，标准评估基础设施（如 Docker 容器和验证脚本）更容易受到模型博弈的影响。我们敦促更广泛的研究社区在设计下一代基准时优先检测和缓解这些行为。

To mitigate reward hacking in coding agent evaluations, we restrict internet access and strip Git histories from the environment. Additionally, we automatically purge transient build and package caches across diverse environments, including Go module caches (go/mod), node modules dependency artifacts, compiled .jar files, and Python pycache directories. Despite these precautions, we still observe instances of exploit-seeking behavior during testing—such as decompiling core Ubuntu Linux packages to uncover vulnerabilities in CyberGym. As models grow increasingly capable, standard evaluation infrastructure (e.g., Docker containers and validation scripts) becomes more susceptible to model gaming. We urge the broader research community to prioritize detecting and mitigating these behaviors when designing next-generation benchmarks.

5.3.2 评估结果 5.3.2 Evaluation Results

如表 3 所示，DeepSeek-V4.1-Flash 在推理和智能体式基准测试中均较其前代 DeepSeek-V4-Flash 展现出显著的性能提升，同时与顶级开源和专有模型持平或超越它们。

As detailed in Table 3, DeepSeek-V4.1-Flash exhibits significant performance upgrades across both reasoning and agentic benchmarks over its predecessor, DeepSeek-V4-Flash, while matching or outperforming top-tier open-source and proprietary models.

在核心推理任务中，DeepSeek-V4.1-Flash 在 Codeforces 上达到 3471 的评分，超越了 DeepSeek-V4-Flash（3289）和 DeepSeek-V4-Pro（3348）。在 MathArena Apex 上，它获得了 65.6% 的 Pass@1 准确率，完全匹配表现最佳的开源基线 Kimi-K3（65.6%）和 DeepSeek-V4-Pro（65.3%）。此外，其 GPQA Diamond 得分达到 90.9%，较 DeepSeek-V4-Flash（89.9%）显示出稳步提升。

In core reasoning tasks, DeepSeek-V4.1-Flash achieves a Codeforces rating of 3471, surpassing both DeepSeek-V4-Flash (3289) and DeepSeek-V4-Pro (3348). On MathArena Apex, it obtains a 65.6% Pass@1 accuracy, fully matching the top-performing open-source baseline Kimi-K3 (65.6%) and DeepSeek-V4-Pro (65.3%). Furthermore, its GPQA Diamond score reaches 90.9%, showing steady improvements over DeepSeek-V4-Flash (89.9%).

在智能体式任务中，性能提升更为显著。值得注意的是，在 DeepSWE v1.1 上，DeepSeek-V4.1-Flash 达到 74.2% 的通过率，较 DeepSeek-V4-Flash（54.4%）大幅跃升，并超越了包括 Opus-5（74.0%）和 GPT-5.6 Sol（73.0%）在内的领先专有模型。在 Terminal-Bench 2.1 上，它达到 90.6%，优于 Opus-5（89.1%）和 GLM-5.3（88.2%）。同样，在 Automation-Bench（54.8%）和 Agents’ Last Exam（31.8%）上，DeepSeek-V4.1-Flash 在开源同类和顶级闭源系统中均取得了领先分数。尽管其规模紧凑，DeepSeek-V4.1-Flash 仍展现出最先进的智能体式能力，大幅缩小了与前沿闭源模型的差距，同时在开源替代方案中确立了明显优势。

The performance gains are even more pronounced across agentic tasks. Notably, on DeepSWE v1.1, DeepSeek-V4.1-Flash reaches 74.2% pass rate, marking a substantial jump from DeepSeek-V4-Flash (54.4%) and surpassing leading proprietary models including Opus-5 (74.0%) and GPT-5.6 Sol (73.0%). On Terminal-Bench 2.1, it achieves 90.6%, outperforming Opus-5 (89.1%) and GLM-5.3 (88.2%). Similarly, on Automation-Bench (54.8%) and Agents’ Last Exam (31.8%), DeepSeek-V4.1-Flash establishes leading scores over both open-source counterparts and top closed-source systems. Despite its compact nature, DeepSeek-V4.1-Flash demonstrates state-of-the-art agentic capabilities, substantially closing the gap with frontier closed-source models while establishing clear advantages among open-source alternatives.

在网络安全任务上，DeepSeek-V4.1-Flash 在开源模型中建立了新的最先进水平。鉴于这些能力的双重用途性质，我们鼓励社区负责任地应用它们，例如用于防御性安全研究和漏洞修复。在视觉智能体任务领域，DeepSeek-V4.1-Flash 展现出强大的能力，特别是在需要视觉推理和复杂专业图表分析的场景中。虽然它超越了领先的开源模型 Kimi-K3，但我们承认，在与领先闭源替代方案进行基准比较时，仍存在可测量的差距。

On cyber-security tasks, DeepSeek-V4.1-Flash establishes a new state of the art among open-source models. Given the dual-use nature of these capabilities, we encourage the community to apply them responsibly, such as for defensive security research and vulnerability remediation. In the domain of visual agent tasks, DeepSeek-V4.1-Flash demonstrates robust capabilities, particularly in scenarios requiring visual reasoning and the analysis of complex professional charts. While it outperforms the leading open-source model Kimi-K3, we acknowledge that a measurable gap still remains when benchmarked against the leading closed-source alternatives.

除了峰值性能，DeepSeek-V4.1-Flash 还提供了一个推理努力程度设置，允许用户以可控的方式在推理成本和准确率之间进行权衡。如图 9 所示，在推理和智能体式基准测试中，准确率和输出长度均随努力程度稳步增加。将努力程度从 25 提高到 100，可使八个推理密集型基准测试的平均 Pass@1 从 67.1% 提升至 76.3%，DeepSWE v1.1 从 66.0% 提升至 74.2%，Terminal-Bench 2.1 从 82.4% 提升至 90.6%，代价是输出 token 大约增加 \(2.5\times)。值得注意的是，在单响应推理上学习到的努力控制能够忠实地迁移到长时程智能体式轨迹中，在那里它控制着跨轮次的探索和验证总量。收益是前置的：60–80 范围已经以不到最大设置一半的 token 预算恢复了其大部分准确率，而最后一步到努力程度 100 仅带来边际改进，却使智能体轨迹延长了 \(1.6) – \(1.8\times)。因此，最高层级最好保留给最具挑战性的任务，而中等努力程度则为日常智能体式使用提供了有利的成本-性能平衡。

Beyond peak performance, DeepSeek-V4.1-Flash exposes a reasoning-effort setting that allows users to trade inference cost for accuracy in a controllable manner. As illustrated in Figure 9, both accuracy and output length increase steadily with the effort level across reasoning and agentic benchmarks alike. Raising the effort from 25 to 100 improves the average Pass@1 on eight reasoning-intensive benchmarks from 67.1% to 76.3%, on DeepSWE v1.1 from 66.0% to 74.2%, and on Terminal-Bench 2.1 from 82.4% to 90.6%, at the cost of roughly \(2.5\times) more output tokens. Notably, the effort control learned on single-response reasoning transfers faithfully to long-horizon agentic trajectories, where it governs the total amount of exploration and verification across turns. The gains are front-loaded: the 60–80 range already recovers most of the accuracy of the maximum setting at less than half of its token budget, whereas the final step to effort 100 lengthens agent trajectories by \(1.6) – \(1.8\times) for only marginal improvements. The maximum tier is thus best reserved for the most challenging tasks, while moderate effort levels offer a favorable cost–performance balance for everyday agentic use.

5.3.4 跨智能体脚手架的性能表现 5.3.4 Performance across agent scaffolds

在实践中，模型很少被部署在单一固定的智能体框架内；不同脚手架在系统提示、工具定义、上下文管理策略和交互协议上各不相同，而一个对特定运行框架过拟合的模型在换到另一个框架时可能性能大幅下降。为评估我们的模型对此类变化的鲁棒性，我们的比较覆盖了来自六个脚手架家族的八种配置：Claude Code、Codex、OpenCode、Pi、mini-SWE 以及 DeepSeek Harness（DSH）的 Minimal、Standard 和 PTC 模式。对于每个脚手架，我们保持模型检查点、解码配置和任务集完全一致，仅改变外围运行框架，包括其原生系统提示、工具模式和轮次交互逻辑。表 4 报告了在 DeepSWE v1.1 和 Terminal-Bench v2.1 上以最大推理努力（100）运行时的性能。

In practice, a model is rarely deployed within a single fixed agent framework; different scaffolds vary in their system prompts, tool definitions, context management strategies, and interaction protocols, and a model that overfits to one particular harness may degrade substantially when placed in another. To assess the robustness of our model to such variation, our comparison covers eight configurations from six scaffold families: Claude Code, Codex, OpenCode, Pi, mini-SWE, and DeepSeek Harness (DSH) in Minimal, Standard, and PTC modes. For each scaffold, we keep the model checkpoint, decoding configuration, and task set identical, and only the surrounding harness, including its native system prompt, tool schema, and turn-taking logic, is changed. Table 4 reports performance at Max reasoning effort (100) on DeepSWE v1.1 and Terminal-Bench v2.1.

注：所有脚手架在 DeepSWE v1.1 上每个任务使用 \(N=8\) 个样本，在 Terminal-Bench v2.1 上使用 \(N=3\) 个样本。运行使用 Linux 容器、温度 1.0、top-p 0.95、1M token 的上下文窗口，并且在两个基准上每个智能体的最大模型生成轮数 max_steps=500。Terminal-Bench v2.1 在无网络访问条件下评估。我们评估了四个 Claude Code 版本，各版本结果及其平均值报告在附录表 5 中；本表报告的是 v2.1.251。更多脚手架特定配置详见附录 B.1。

Note. All scaffolds use \(N=8\) samples per task on DeepSWE v1.1 and \(N=3\) on Terminal-Bench v2.1. Runs use Linux containers, temperature 1.0, top-p 0.95, a 1M-token context window, and max_steps=500 model-generation rounds per agent on both benchmarks. Terminal-Bench v2.1 is evaluated without network access. We evaluate four Claude Code versions, with per-version results and their average reported in Appendix Table 5; this table reports v2.1.251. More scaffold-specific configurations are detailed in Appendix B.1.

该模型的智能体能力在不同提示和工具接口的脚手架家族之间迁移良好，而非依赖于特定运行框架的约定。随着外围交互协议和工具抽象的变化，其性能保持稳健，表明其智能体行为并未与单一脚手架设计紧密耦合。这种鲁棒性与我们合成训练数据（第 5 节）中环境、工具模式和交互格式的多样性一致，该数据旨在鼓励跨智能体脚手架的泛化。

The model’s agentic capabilities transfer well across scaffold families with different prompts and tool interfaces, rather than depending on conventions specific to a particular harness. Its performance remains robust as the surrounding interaction protocol and tool abstractions change, indicating that its agentic behavior is not tightly coupled to a single scaffold design. This robustness is consistent with the diversity of environments, tool schemas, and interaction formats in our synthesized training data (Section 5), which is designed to encourage generalization across agent scaffolds.

5.3.5 多智能体 5.3.5 Multi-Agent

为探索复杂任务上的多智能体协作，我们使用 DeepSeek Harness 的 Agent Team 模式对 DeepSeek-V4.1-Flash 进行了初步实验。

To explore multi-agent collaboration on complex tasks, we conduct preliminary experiments with DeepSeek-V4.1-Flash using DeepSeek Harness's Agent Team mode.

**多智能体运行框架。** 我们使用 DeepSeek Harness 的 Agent Team 模式，其中主导智能体（lead agent）默认可通过 `spawn_teammate` 异步创建具名、持久的队友。每个队友接收一项委派任务，并以两种模式之一启动：fresh 模式（不含主导智能体的历史）或 fork 模式（携带主导智能体已完成轮次的一次性快照）。所有智能体共享同一份代码仓库检出，因此彼此的编辑立即可见。

**Multi-Agent Harnesses.** We use DeepSeek Harness in Agent Team mode, where a lead agent can asynchronously create named, persistent teammates by default through `spawn_teammate`. Each receives a delegated task and starts in either fresh mode without lead history or fork mode with a one-time snapshot of the lead's completed turns. All agents share one repository checkout, making edits immediately visible to one another.

智能体通过持久化的对等邮箱（peer mailbox）进行通信。通过 `send_message` 发送的消息会在运行中的队友到达下一步边界时送达，为闲置队友开启新一轮，或唤醒不活跃的队友。在每个队友的任务生命周期内，主导智能体通过 `list_agents` 监控运行时状态，并通过 `wait_agent` 等待状态、邮箱或共享任务的变化。任务归属、依赖关系以及建议性写入范围维护在共享任务板上（使用四个 `team_task_*` 工具，更新时进行修订检查）。当需要干预时，只有主导智能体可以通过 `interrupt_agent` 中断队友当前轮次。所需工作完成后，主导智能体审查并测试合并后的改动，并生成最终回复。

Agents communicate through a durable peer mailbox. A message sent through `send_message` reaches a running teammate at its next step boundary, starts a new turn for an idle teammate, or resumes an inactive teammate. Across each teammate's task lifecycle, the lead monitors runtime status with `list_agents` and waits for status, mailbox, or shared-task changes with `wait_agent`. Task ownership, dependencies, and advisory write scopes are maintained on a shared task board (using the four `team_task_*` tools with revision checks on updates). When intervention is needed, only the lead can interrupt a teammate's current turn through `interrupt_agent`. Once the required work is complete, the lead reviews and tests the combined changes and produces the final response.

**训练。** 我们以强化学习奖励训练 Agent Team 模式，该奖励结合了任务表现、鼓励委派与智能体间通信的协作奖励，以及促进高效协调的派生延迟惩罚。派生延迟的计算方式为：将执行事件及其协作依赖表示为有向无环图（DAG），按固定 prefill/decode 速率由 token 数加上实测工具执行时间分配成本，并取关键路径的长度。这鼓励有益的并行，同时惩罚不必要的串行工作与同步，并降低对服务端批处理与排队延迟的敏感性。

**Training.** We train Agent Team mode with an RL reward combining task performance, a collaboration bonus that encourages delegation and inter-agent communication, and a derived-latency penalty that promotes efficient coordination. Derived latency is computed by representing execution events and their collaboration dependencies as a directed acyclic graph (DAG), assigning costs from token counts at fixed prefill/decode rates plus measured tool-execution time, and taking the length of the critical path. This encourages useful parallelism while penalizing unnecessary sequential work and synchronization, with reduced sensitivity to serving-side batching and queuing delays.

**性能。** 我们通过仅保留参考解在隐藏测试集上通过率至少达到 95% 的任务，构建了 ProgramBench 的高置信度子集。该过滤过程留下 172 个“黄金”任务。我们还在 FrontierSWE v2 上评估，它是 FrontierSWE 的更大、更具挑战性的后继版本，采用了大幅改进的方法论。我们通过排除需要 GPU 访问的任务，从当前公开任务中构建了一个无 GPU 子集。我们在明确的每次 rollout 墙钟时间截止条件下，在两个基准上评估单智能体与多智能体配置。此处报告的结果为初步结果：我们将观察到的最强多智能体配置与可用的最强单智能体基线进行比较。在 ProgramBench 上，我们对每个任务最多运行三次 rollout，即每个配置对应 516 次计划 rollout。我们报告 Almost@1，它衡量得分至少达到 0.95 的单个 rollout 所占比例。在 FrontierSWE v2 上，我们报告 Mean@5。ProgramBench 的截止时间为 1 至 12 小时，而 FrontierSWE v2 在 1 至 20 小时的截止时间下评估。在每个截止时间点，指标均根据截止时间到达时可用的输出计算。

**Performance.** We construct a high-confidence subset of ProgramBench by retaining only tasks for which the reference solution achieves a pass rate of at least 95% on the hidden test suite. This filtering procedure leaves 172 "golden" tasks. We also evaluate on FrontierSWE v2, a larger and more challenging successor to FrontierSWE that uses a substantially improved methodology. We construct a no-GPU subset from the currently public tasks by excluding tasks that require GPU access. We evaluate single-agent and multi-agent configurations on both benchmarks under explicit per-rollout wall-clock deadlines. The results reported here are preliminary: we compare the strongest observed multi-agent configurations with the strongest available single-agent baselines. On ProgramBench, we run up to three rollouts per task, corresponding to 516 planned rollouts for each configuration. We report Almost@1, which measures the fraction of individual rollouts achieving a score of at least 0.95. On FrontierSWE v2, we report Mean@5. ProgramBench deadlines range from 1 to 12 hours, while FrontierSWE v2 is evaluated at deadlines ranging from 1 to 20 hours. At each deadline, metrics are computed from the outputs available when the deadline is reached.

如图 10 所示，在两个基准测试上，多智能体配置在每个截止时间点都优于单智能体配置。在 ProgramBench 上，多智能体配置的 Almost@1 从 1 小时时的 13.59% 提升至 8 小时时的峰值 30.04%，而单智能体配置则为 12.79% 和 20.39%。在 FrontierSWE v2 上，多智能体配置的 Mean@5 从 1 小时时的 13.50% 提升至 20 小时时的 32.90%，而单智能体配置则从 10.50% 提升至 28.20%。

As shown in Figure 10, multi-agent configurations outperform their single-agent counterparts at every deadline on both benchmarks. On ProgramBench, Almost@1 increases from 13.59% at 1 hour to a peak of 30.04% at 8 hours for the multi-agent configuration, compared with 12.79% and 20.39% for the single-agent configuration. On FrontierSWE v2, Mean@5 increases from 13.50% at 1 hour to 32.90% at 20 hours for the multi-agent configuration, compared with an increase from 10.50% to 28.20% for the single-agent configuration.

6 结论、局限性与未来方向 6 Conclusion, Limitations, and Future Directions

在本工作中，我们提出了 DeepSeek-V4.1-Flash，一个支持高达一百万 token 上下文的多模态混合专家（MoE）模型。通过对模型架构、缓存精度和部署策略的联合优化，DeepSeek-V4.1-Flash 将 KV 缓存压缩推向了极限。其因果编码器-解码器（CED）架构使模型在预填充阶段每个 token 仅激活 8B 参数，而在解码阶段激活 16B，从而提升了输入密集型智能体式工作负载的成本效率。在相同序列长度下，压缩稀疏注意力机制 2（CSA2）中的跨层 KV 缓存复用与 FP4 KV 缓存将其全局 KV 缓存占用（始终位于 HBM）降至每 token 890 字节，约为 DeepSeek-V4-Flash 相应占用的 1/4。SWA 有界重放进一步将其持久化 KV 缓存占用（始终位于 SSD 或主机内存）降至 DeepSeek-V4-Flash 的约 1/8。这些缩减缓解了 HBM 和 SSD 的容量压力，同时该模型整体性能显著优于 DeepSeek-V4-Flash。尽管参数规模远小于 GLM-5.3 和 Kimi-K3 等当代开源模型，DeepSeek-V4.1 在关键基准测试上仍取得了相当——在若干任务上甚至更优——的性能。

In this work, we introduce DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model with support for contexts of up to one million tokens. Through joint optimization of model architecture, cache precision, and deployment strategy, DeepSeek-V4.1-Flash pushes the limits of KV cache compression. Its Causal Encoder-Decoder (CED) architecture enables the model to activate only 8B parameters per token during prefill, compared with 16B during decode, improving cost efficiency for input-heavy agentic workloads. At equal sequence lengths, cross-layer KV cache reuse in Compressed Sparse Attention 2 (CSA2) and FP4 KV caching reduce its global KV cache footprint (always in HBM) to 890 bytes per token, roughly 1/4 of the corresponding footprint of DeepSeek-V4-Flash. SWA Bounded Replay further reduces its persistent KV cache footprint (always on SSD or in host memory) to roughly 1/8 of that of DeepSeek-V4-Flash. These reductions alleviate HBM and SSD capacity pressure while the model delivers substantially better overall performance than DeepSeek-V4-Flash. Despite possessing a significantly smaller parameter footprint than contemporary open-source models such as GLM-5.3 and Kimi-K3, DeepSeek-V4.1 achieves comparable—and in several tasks, superior—performance across key benchmarks.

尽管 DeepSeek-V4.1-Flash 相对于 DeepSeek-V4-Flash 大幅简化了若干架构组件，但新引入的架构改动也带来了尚未被充分刻画的鲁棒性边界。我们的内部评估覆盖了多样化的测试用例和边界条件，在评估设置中未观察到模型能力的系统性退化。然而，任何有限的测试集都无法覆盖所有极端输入和部署条件。CSA2 中潜在的选择错误以及 SWA 有界重放中的近似状态重建，仍可能在未测试的边界情况下导致能力退化。未来，我们将继续扩展压力测试与评估体系，特别关注长上下文下的稀疏检索以及缓存恢复边界处的 SWA 状态重建。我们还将监控真实工作负载，系统性地刻画潜在失效模式与鲁棒性边界，并进一步提升模型在极端条件下的鲁棒性。

Although DeepSeek-V4.1-Flash substantially simplifies several architectural components relative to DeepSeek-V4-Flash, the newly introduced architectural changes also create robustness boundaries that have yet to be fully characterized. Our internal evaluations cover a diverse range of test cases and boundary conditions, and we have not observed any systematic degradation in model capabilities in the evaluated settings. Nevertheless, no finite test suite can cover every extreme input and deployment condition. Potential selection errors in CSA2 and approximate state reconstruction in SWA Bounded Replay may still cause capability degradation in untested boundary cases. Going forward, we will continue to expand our stress-testing and evaluation stack, with particular attention to sparse retrieval over long contexts and SWA state reconstruction at cache-resumption boundaries. We will also monitor real-world workloads, systematically characterize potential failure modes and robustness boundaries, and further improve model robustness under extreme conditions.

随着 AI 模型取得卓越的性能表现，标准评估基准日益趋于饱和。尽管 DeepSeek-V4.1-Flash 展现出接近 Fable-5 和 GPT-6 Astra 等顶级模型的性能——在日常应用中提供高度可比的用户体验——但在最具挑战性的任务上仍存在性能差距。虽然基准分数差距不大，但这种持平并不意味着该模型在复杂、高难度推理和边缘案例上已具备领先闭源系统的前沿能力。

As AI models achieve remarkable performance capabilities, standard evaluation benchmarks have increasingly reached saturation. While DeepSeek-V4.1-Flash demonstrates performance that closely approaches top-tier models like Fable-5 and GPT-6 Astra—offering a highly comparable user experience in daily applications—a performance gap remains on the most challenging tasks. Although benchmark scores show a narrow margin, this parity does not imply that the model matches the frontier capabilities of leading closed-source systems on complex, high-difficulty reasoning and edge cases.

因此，我们将持续更新评估协议，以确保对最先进推理边界进行严格评估。在持续降低模型成本的同时，我们相信模型智能的进一步进步将依赖于数据、模型容量和强化学习的协同规模扩张。以 DeepSeek-V4.1-Flash 为新起点，我们将继续探索模型能力的极限，并系统性地应对大规模数据合成和强化学习规模扩张中的关键挑战。我们还将积极整合模型与工具链的协同设计，使联合系统能够共同演进和优化。通过同步推进成本降低与能力规模扩张，我们希望让高能力智能体更易获取、更易部署，进一步降低 AI 技术在更广泛行业和场景中的采用门槛。

Consequently, we will continuously update our evaluation protocols to ensure rigorous assessment of state-of-the-art reasoning boundaries. Alongside continued efforts to reduce model costs, we believe that further advances in model intelligence will depend on the coordinated scaling of data, model capacity, and RL. With DeepSeek-V4.1-Flash as a new starting point, we will continue to explore the limits of model capabilities and systematically address key challenges in large-scale data synthesis and RL scaling. We will also actively integrate model–harness co-design, enabling the joint system to evolve and be optimized together. By advancing cost reduction and capability scaling in tandem, we hope to make highly capable agents more accessible and easier to deploy, further lowering the barriers to adopting AI technologies across a broader range of industries and scenarios.