In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top-$K$ out of $N$ experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into $mN$ ones and activating $mK$ from them, allowing for a more flexible combination of activated experts; (2) isolating $K_s$ experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5 times the expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations. Further, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.
核心贡献 · Key contributions
提出 DeepSeekMoE 架构，通过细粒度专家分割和共享专家隔离实现极致专家专业化。 Proposes DeepSeekMoE architecture with fine-grained expert segmentation and shared expert isolation for ultimate expert specialization.
实验验证 DeepSeekMoE 2B 几乎达到 MoE 模型的上限性能。 Empirically validates that DeepSeekMoE 2B nearly approaches the upper bound performance of MoE models.
将 DeepSeekMoE 扩展到 16B，仅用约 40%算力达到与 LLaMA2 7B 相当的性能。 Scales DeepSeekMoE to 16B, achieving comparable performance with LLaMA2 7B using only about 40% computations.
通过对 DeepSeekMoE 16B 进行监督微调成功实现对齐，创建了有效的聊天模型。 Demonstrates successful alignment via supervised fine-tuning on DeepSeekMoE 16B, creating an effective chat model.
初步扩展到 145B 显示了对 GShard 的持续优势，并使用 28.5%算力达到与 DeepSeek 67B 相当的性能。 Preliminary scaling to 145B shows consistent advantages over GShard and comparable performance with DeepSeek 67B using 28.5% computations.
公开发布 DeepSeekMoE 16B 检查点，可在单个 40GB GPU 上无需量化部署。 Releases DeepSeekMoE 16B checkpoint publicly, deployable on a single 40GB GPU without quantization.
局限 · Limitations
验证实验限于 2B 参数和 100B token，可能无法推广到更大规模。 Validation experiments limited to 2B parameters and 100B tokens, may not generalize to larger scales.
DeepSeekMoE 16B 由于注意力参数较少，在多项选择任务中表现有限。 DeepSeekMoE 16B shows limitations in multiple-choice tasks due to fewer attention parameters.
初步 145B 研究仅在 245B token 上训练，未完全优化。 Preliminary 145B study trained only on 245B tokens, not fully optimized.
共享专家隔离可能并非对所有任务最优；共享与路由专家比例需要调整。 Shared expert isolation may not be optimal for all tasks; ratio of shared to routed experts requires tuning.
评估主要在英文和中文基准上；未评估其他语言性能。 Evaluation primarily on English and Chinese benchmarks; performance on other languages not assessed.
论文章节 · Sections（共 23）
摘要Abstract
1 引言1 Introduction
2 预备知识：面向 Transformer 的混合专家模型2 Preliminaries: Mixture-of-Experts for Transformers
3 DeepSeekMoE 架构3 DeepSeekMoE Architecture
3.1 细粒度专家分割3.1 Fine-Grained Expert Segmentation
3.2 共享专家隔离3.2 Shared Expert Isolation
3.3 负载均衡考虑3.3 Load Balance Consideration
4.1 实验设置4.1 Experimental Setup
4.2 评估4.2 Evaluations
4.3 DeepSeekMoE 与 MoE 模型上界紧密对齐4.3 DeepSeekMoE Aligns Closely with the upper bound of MoE Models
4.4 消融研究4.4 Ablation Studies
4.5 专家专业化分析4.5 Analysis on Expert Specialization
5 扩展至 DeepSeekMoE 16B5 Scaling up to DeepSeekMoE 16B
5.1 实验设置5.1 Experimental Setup
5.2 评估5.2 Evaluations
6 DeepSeekMoE 16B 的对齐6 Alignment for DeepSeekMoE 16B