DeepSeek-Coder-V2：打破闭源模型在代码智能领域的壁垒

DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

深度求索 DeepSeek-AI · DeepSeek · 2024-06-17 · arXiv:2406.11931 ↗ · 被引 455

摘要 · Abstract

我们推出了 DeepSeek-Coder-V2，这是一个开源的混合专家（MoE）代码语言模型，在代码特定任务上实现了与 GPT4-Turbo 相当的性能。具体来说，DeepSeek-Coder-V2 从 DeepSeek-V2 的一个中间检查点开始，额外预训练了 6 万亿个 token。通过这种持续预训练，DeepSeek-Coder-V2 显著增强了 DeepSeek-V2 的编码和数学推理能力，同时保持了一般语言任务的性能。与 DeepSeek-Coder-33B 相比，DeepSeek-Coder-V2 在代码相关任务、推理和通用能力方面都取得了显著进步。此外，DeepSeek-Coder-V2 将支持的编程语言从 86 种扩展到 338 种，并将上下文长度从 16K 扩展到 128K。在标准基准评估中，DeepSeek-Coder-V2 在编码和数学基准测试中取得了优于 GPT4-Turbo、Claude 3 Opus 和 Gemini 1.5 Pro 等闭源模型的性能。

We present DeepSeek-Coder-V2, an open-source Mixture-of-Experts (MoE) code language model that achieves performance comparable to GPT4-Turbo in code-specific tasks. Specifically, DeepSeek-Coder-V2 is further pre-trained from an intermediate checkpoint of DeepSeek-V2 with additional 6 trillion tokens. Through this continued pre-training, DeepSeek-Coder-V2 substantially enhances the coding and mathematical reasoning capabilities of DeepSeek-V2, while maintaining comparable performance in general language tasks. Compared to DeepSeek-Coder-33B, DeepSeek-Coder-V2 demonstrates significant advancements in various aspects of code-related tasks, as well as reasoning and general capabilities. Additionally, DeepSeek-Coder-V2 expands its support for programming languages from 86 to 338, while extending the context length from 16K to 128K. In standard benchmark evaluations, DeepSeek-Coder-V2 achieves superior performance compared to closed-source models such as GPT4-Turbo, Claude 3 Opus, and Gemini 1.5 Pro in coding and math benchmarks.

核心贡献 · Key contributions

开源 MoE 代码模型，在代码任务上达到 GPT4-Turbo 级别性能。
Open-source MoE code model achieving GPT4-Turbo-level performance in code tasks.
在 6 万亿 token 上继续预训练，增强了编码和数学推理能力，同时保持通用语言性能。
Continued pre-training on 6T tokens enhances coding and math reasoning while maintaining general language performance.
将编程语言支持从 86 种扩展到 338 种，上下文长度从 16K 扩展到 128K。
Expands programming language support from 86 to 338 and context length from 16K to 128K.
在编码和数学基准上优于 GPT4-Turbo、Claude 3 Opus 和 Gemini 1.5 Pro 等闭源模型。
Outperforms closed-source models like GPT4-Turbo, Claude 3 Opus, and Gemini 1.5 Pro on coding and math benchmarks.
首个在 SWEBench 上超过 10%的开源模型，在 HumanEval 上达到 90.2%。
First open-source model to exceed 10% on SWEBench, achieving 90.2% on HumanEval.
使用 GRPO 进行对齐，利用编译器反馈和奖励模型优化人类偏好。
Uses GRPO for alignment with human preferences, leveraging compiler feedback and reward models.

局限 · Limitations

与 GPT-4 Turbo 相比，指令遵循能力存在显著差距，导致在 SWEbench 等复杂任务上表现不佳。
Significant gap in instruction-following capabilities compared to GPT-4 Turbo, leading to poor performance on complex tasks like SWEbench.
预训练数据截止于 2023 年 11 月前，可能限制对近期事件的了解。
Pre-training data cutoff before November 2023 may limit knowledge of recent events.
在 TriviaQA 等知识密集型基准上，自然语言性能略逊于 DeepSeek-V2。
Natural language performance slightly lags behind DeepSeek-V2 on knowledge-intensive benchmarks like TriviaQA.
236B 模型仅使用下一个词预测目标，未使用 Fill-In-Middle，限制了代码补全能力。
236B model uses only next-token prediction objective, not Fill-In-Middle, limiting code completion capabilities.
由于指数归一化导致训练不稳定，需回退到传统方法。
Training instability observed due to exponential normalization, requiring reversion to conventional method.

论文章节 · Sections（共 18）

摘要 Abstract
1 引言 1 Introduction
1.1 贡献 1.1 Contributions
1.2 评估与指标概述 1.2 Summary of Evaluations and Metrics
2 数据收集 2 Data Collection
3.1 训练策略 3.1 Training Strategy
3.2 模型架构 3.2 Model Architecture
3.3 训练超参数 3.3 Training Hyper-Parameters
3.4 长上下文扩展 3.4 Long Context Extension
3.5 对齐 3.5 Alignment
4 实验结果 4 Experimental Results
4.1 代码生成 4.1 Code Generation
4.2 代码补全 4.2 Code Completion
4.3 代码修复 4.3 Code Fixing
4.4 代码理解与推理 4.4 Code Understanding and Reasoning
4.5 数学推理 4.5 Mathematical Reasoning
4.6 通用自然语言 4.6 General Natural Language
5 结论 5 Conclusion

阅读逐段中英对照全文 →