DeepSeek-V4：迈向高效百万令牌上下文智能

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

深度求索 DeepSeek-AI · · 2026-04-26 · arXiv:2606.19348 ↗ · 被引 236

摘要 · Abstract

我们推出了 DeepSeek-V4 系列的预览版本，包括两个强大的混合专家（MoE）语言模型——参数为 1.6T（激活 49B）的 DeepSeek-V4-Pro 和参数为 284B（激活 13B）的 DeepSeek-V4-Flash，两者均支持百万令牌的上下文长度。DeepSeek-V4 系列在架构和优化方面引入了多项关键升级：（1）混合注意力架构，结合压缩稀疏注意力（CSA）和高度压缩注意力（HCA），提升长上下文效率；（2）流形约束超连接（mHC），增强传统残差连接；（3）Muon 优化器，实现更快的收敛和更高的训练稳定性。我们在超过 32T 多样化和高质量的令牌上预训练了这两个模型，随后进行了全面的后训练流程，以解锁并进一步增强它们的能力。DeepSeek-V4-Pro-Max 是 DeepSeek-V4-Pro 的最大推理努力模式，重新定义了开放模型的最高水平，在核心任务上超越了其前代模型。同时，DeepSeek-V4 系列在长上下文场景中表现出极高的效率。在百万令牌上下文设置下，与 DeepSeek-V3.2 相比，DeepSeek-V4-Pro 仅需 27%的单令牌推理 FLOPs 和 10%的 KV 缓存。这使得我们能够常规支持百万令牌上下文，从而使长期任务和进一步的测试时扩展更加可行。模型检查点可在 https://huggingface.co/collections/deepseek-ai/deepseek-v4 获取。

We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models -- DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated) -- both supporting a context length of one million tokens. DeepSeek-V4 series incorporate several key upgrades in architecture and optimization: (1) a hybrid attention architecture that combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to improve long-context efficiency; (2) Manifold-Constrained Hyper-Connections (mHC) that enhance conventional residual connections; (3) and the Muon optimizer for faster convergence and greater training stability. We pre-train both models on more than 32T diverse and high-quality tokens, followed by a comprehensive post-training pipeline that unlocks and further enhances their capabilities. DeepSeek-V4-Pro-Max, the maximum reasoning effort mode of DeepSeek-V4-Pro, redefines the state-of-the-art for open models, outperforming its predecessors in core tasks. Meanwhile, DeepSeek-V4 series are highly efficient in long-context scenarios. In the one-million-token context setting, DeepSeek-V4-Pro requires only 27% of single-token inference FLOPs and 10% of KV cache compared with DeepSeek-V3.2. This enables us to routinely support one-million-token contexts, thereby making long-horizon tasks and further test-time scaling more feasible. The model checkpoints are available at https://huggingface.co/collections/deepseek-ai/deepseek-v4.

核心贡献 · Key contributions

混合注意力（CSA+HCA）大幅降低 KV 缓存和算力；百万 token 上下文仅需 DeepSeek-V3.2 的 27%算力。
Hybrid attention (CSA+HCA) reduces KV cache and FLOPs; 1M-token context requires only 27% FLOPs of DeepSeek-V3.2.
流形约束超连接（mHC）提升训练稳定性与表达能力，超越标准残差连接。
Manifold-Constrained Hyper-Connections (mHC) stabilize training and improve expressivity beyond standard residuals.
Muon 优化器在超大规模 MoE 模型上实现更快收敛与稳定训练。
Muon optimizer enables faster convergence and stable training for MoE models at scale.
后训练管道采用领域专用强化学习与在线策略蒸馏，统一多领域专家模型。
Post-training pipeline uses domain-specific RL and on-policy distillation to unify expert models.
DeepSeek-V4-Pro-Max 在推理、编码、长上下文任务中达到开放模型最先进水平。
DeepSeek-V4-Pro-Max achieves SOTA among open models on reasoning, coding, and long-context tasks.
高效百万 token 上下文使长期任务与测试时规模扩展变得可行。
Efficient 1M-token context enables long-horizon tasks and test-time scaling previously infeasible.

局限 · Limitations

知识基准仍落后于领先专有模型如 Gemini-3.1-Pro。
Knowledge benchmarks still trail leading proprietary models like Gemini-3.1-Pro.
智能体能力在复杂任务上不及前沿闭源模型。
Agent capabilities fall short of frontier closed models on complex tasks.
FP4 量化效率依赖于尚未可用的未来硬件。
FP4 quantization efficiency depends on future hardware not yet available.
混合注意力因压缩逻辑可能在短上下文中引入开销。
Hybrid attention may introduce overhead for short-context scenarios due to compression logic.
MoE 与注意力设计在大规模下带来复杂的通信与存储管理。
MoE and attention designs impose complex communication and storage management at scale.

论文章节 · Sections（共 47）

DeepSeek-V4：迈向高效百万词元上下文智能 DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
1 引言 1 Introduction
2 架构 2 Architecture
2.1 继承自 DeepSeek-V3 的设计 2.1 Designs Inherited from DeepSeek-V3
2.2 流形约束超连接 2.2 Manifold-Constrained Hyper-Connections
2.3 混合注意力机制：CSA 与 HCA 2.3 Hybrid Attention with CSA and HCA
2.3.1 压缩稀疏注意力 2.3.1 Compressed Sparse Attention
2.3.2 重度压缩注意力 2.3.2 Heavily Compressed Attention
2.3.3 其他细节 2.3.3 Other Details
2.3.4 效率讨论 2.3.4 Efficiency Discussion
2.4 Muon 优化器 2.4 Muon Optimizer
3.1 专家并行中的细粒度通信计算重叠 3.1 Fine-Grained Communication-Computation Overlap in Expert Parallelism
3.2 使用 TileLang 灵活高效的内核开发 3.2 Flexible and Efficient Kernel Development with TileLang
3.3 高性能批次不变且确定性的内核库 3.3 High-Performance Batch-Invariant and Deterministic Kernel Libraries
3.4 训练框架 3.4 Training Framework
3.4.1 Muon 的高效实现 3.4.1 Efficient Implementation of Muon
3.4.2 mHC 的成本高效和内存高效实现 3.4.2 Cost-Effective and Memory-Efficient Implementation of mHC
3.4.3 长上下文注意力的上下文并行 3.4.3 Contextual Parallelism for Long-Context Attention
3.4.4 扩展自动微分以实现灵活的激活检查点 3.4.4 Extended Automatic Differentiation for Flexible Activation Checkpointing
3.5 推理框架 3.5 Inference Framework
3.5.1 KV 缓存结构与管理 3.5.1 KV Cache Structure and Management
3.5.2 基于磁盘的 KV 缓存存储 3.5.2 On-Disk KV Cache Storage
4.1 数据构建 4.1 Data Construction
4.2.1 模型设置 4.2.1 Model Setups
4.2.2 训练设置 4.2.2 Training Setups
4.2.3 缓解训练不稳定性 4.2.3 Mitigating Training Instability
4.3.1 评估基准 4.3.1 Evaluation Benchmarks
4.3.2 评估结果 4.3.2 Evaluation Results
5.1 后训练流程 5.1 Post-Training Pipeline
5.1.1 专家训练 5.1.1 Specialist Training
5.1.2 在策略蒸馏 5.1.2 On-Policy Distillation
5.2 后训练基础设施 5.2 Post-Training Infrastructures
5.2.1 FP4 量化感知训练 5.2.1 FP4 Quantization-Aware Training
5.2.2 面向全词汇 OPD 的高效教师调度 5.2.2 Efficient Teacher Scheduling for Full-Vocabulary OPD
5.2.3 可抢占且容错的推出服务 5.2.3 Preemptible and Fault-Tolerant Rollout Service
5.2.4 扩展百万 Token 上下文强化学习框架 5.2.4 Scaling RL Framework for Million-Token Context
5.2.5 智能体式人工智能的沙盒基础设施 5.2.5 Sandbox Infrastructure for Agentic AI
5.3.1 评估设置 5.3.1 Evaluation Setup
5.3.2 评估结果 5.3.2 Evaluation Results
5.4 真实世界任务性能 5.4 Performance on Real-World Tasks
5.4.1 中文写作 5.4.1 Chinese Writing
5.4.2 搜索 5.4.2 Search
5.4.3 白领任务 5.4.3 White-Collar Task
5.4.4 代码智能体 5.4.4 Code Agent
6 结论、局限与未来方向 6 Conclusion, Limitations, and Future Directions
A.1 作者列表 A.1 Author List
致谢 A.2 Acknowledgment

阅读逐段中英对照全文 →