DeepSeek-V3 技术报告

DeepSeek-V3 Technical Report

梁文锋 Wenfeng Liang · DeepSeek · 2024-12-27 · arXiv:2412.19437 ↗ · 被引 0

摘要 · Abstract

我们提出了 DeepSeek-V3，一个强大的混合专家（MoE）语言模型，总参数量为 671B，每个 token 激活 37B 参数。为了实现高效推理和成本效益训练，DeepSeek-V3 采用了多头潜在注意力（MLA）和 DeepSeekMoE 架构，这些架构在 DeepSeek-V2 中得到了充分验证。此外，DeepSeek-V3 首创了无辅助损失的负载均衡策略，并设置了多 token 预测训练目标以提升性能。我们在 14.8 万亿多样化且高质量的 token 上预训练了 DeepSeek-V3，随后进行了监督微调和强化学习阶段以充分发挥其能力。全面评估显示，DeepSeek-V3 优于其他开源模型，并达到了与领先闭源模型相当的性能。尽管性能卓越，DeepSeek-V3 的完整训练仅需 2.788M H800 GPU 小时。此外，其训练过程非常稳定。在整个训练过程中，我们没有遇到任何不可恢复的损失尖峰或进行任何回滚。模型检查点可在 https://github.com/deepseek-ai/DeepSeek-V3 获取。

We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks. The model checkpoints are available at https://github.com/deepseek-ai/DeepSeek-V3.

核心贡献 · Key contributions

首创无辅助损失的 MoE 负载均衡策略，最小化性能下降。
Pioneers an auxiliary-loss-free load balancing strategy for MoE, minimizing performance degradation.
引入多词预测训练目标，提升整体基准性能。
Introduces multi-token prediction (MTP) training objective, enhancing overall benchmark performance.
首次在超大规模模型上验证 FP8 混合精度训练的有效性。
Validates FP8 mixed precision training on an extremely large-scale model for the first time.
设计 DualPipe 算法实现高效流水线并行，通信开销近乎为零。
Designs DualPipe algorithm for efficient pipeline parallelism with near-zero communication overhead.
在开源模型中达到最先进性能，与领先闭源模型相当。
Achieves state-of-the-art performance among open-source models, comparable to leading closed-source models.
全训练仅需 278.8 万 H800 GPU 小时，成本效益卓越。
Requires only 2.788M H800 GPU hours for full training, demonstrating exceptional cost efficiency.

局限 · Limitations

训练成本不包括架构、算法或数据的先前研究和消融实验。
Training costs exclude prior research and ablation experiments on architectures, algorithms, or data.
英语事实知识（SimpleQA）性能落后于 GPT-4o 和 Claude-Sonnet-3.5。
Performance on English factual knowledge (SimpleQA) trails behind GPT-4o and Claude-Sonnet-3.5.
工程相关任务略低于 Claude-Sonnet-3.5，但仍优于其他模型。
Engineering-related tasks slightly below Claude-Sonnet-3.5, though still outperforming other models.
推理时丢弃 MTP 模块，仅用于训练改进。
MTP modules are discarded during inference; only used for training improvement.
多行提示无终止换行时可能出现词边界偏差。
Token boundary bias may occur for multi-line prompts without terminal line breaks.

论文章节 · Sections（共 20）

摘要 Abstract
1 引言 1 Introduction
2 架构 2 Architecture
2.1 基本架构 2.1 Basic Architecture
2.2 多令牌预测 2.2 Multi-Token Prediction
3.1 计算集群 3.1 Compute Clusters
3.2 训练框架 3.2 Training Framework
3.3 FP8 训练 3.3 FP8 Training
3.4 推理与部署 3.4 Inference and Deployment
3.5 硬件设计建议 3.5 Suggestions on Hardware Design
4.1 数据构建 4.1 Data Construction
4.2 超参数 4.2 Hyper-Parameters
4.3 长上下文扩展 4.3 Long Context Extension
4.4 评估 4.4 Evaluations
4.5 讨论 4.5 Discussion
5.1 监督微调 5.1 Supervised Fine-Tuning
5.2 强化学习 5.2 Reinforcement Learning
5.3 评估 5.3 Evaluations
5.4 讨论 5.4 Discussion
6 结论、局限与未来方向 6 Conclusion, Limitations, and Future Directions

阅读逐段中英对照全文 →