We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.
核心贡献 · Key contributions
提出多头潜在注意力（MLA），通过低秩键值联合压缩实现高效推理。 Proposes Multi-head Latent Attention (MLA) with low-rank KV joint compression for efficient inference.
采用 DeepSeekMoE，通过细粒度专家分割和共享专家隔离实现经济训练。 Adopts DeepSeekMoE with fine-grained expert segmentation and shared expert isolation for economical training.
仅用 210 亿激活参数即在开源模型中达到顶级性能。 Achieves top-tier performance among open-source models with only 21B activated parameters.
相比 DeepSeek 67B 节省 42.5%训练成本，减少 93.3%键值缓存，提升生成吞吐量 5.76 倍。 Saves 42.5% training costs, reduces KV cache by 93.3%, and boosts generation throughput by 5.76x vs DeepSeek 67B.
通过 YaRN 扩展支持 128K 上下文长度，并展示稳健的长上下文性能。 Supports 128K context length via YaRN extension and demonstrates robust long-context performance.
采用两阶段强化学习（推理对齐后人类偏好对齐），在不牺牲标准基准下增强对齐。 Employs two-stage RL (reasoning then human preference) to enhance alignment without sacrificing standard benchmarks.
局限 · Limitations
预训练后缺乏持续知识更新，限制了对新信息的适应能力。 Lacks ongoing knowledge updates after pre-training, limiting adaptability to new information.
可能生成非事实信息或幻觉，尤其在未经验证的建议场景中。 May generate non-factual information or hallucinations, especially in unverified advice scenarios.
由于数据构成，中英文以外的语言能力有限。 Proficiency in languages other than Chinese and English is limited due to data composition.
观察到对齐税：强化学习训练略微降低了 BBH 等标准基准的性能。 Alignment tax observed: RL training slightly degrades performance on some standard benchmarks like BBH.
目前仅支持文本模态，多模态能力尚未实现。 Currently supports text modality only; multi-modal capabilities are not yet available.