DeepSeek-Coder：当大型语言模型遇见编程——代码智能的崛起

DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

郭达雅 Daya Guo · DeepSeek · 2024-01-25 · arXiv:2401.14196 ↗ · 被引 1735

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

大型语言模型的快速发展彻底改变了软件开发中的代码智能。然而，闭源模型的主导地位限制了广泛的研究和开发。为了解决这一问题，我们推出了 DeepSeek-Coder 系列，这是一系列开源代码模型，参数规模从 1.3B 到 33B，从头开始在 2 万亿个 token 上训练。这些模型在高质量的项目级代码语料库上进行预训练，并采用具有 16K 窗口的填空任务来增强代码生成和填充。我们的广泛评估表明，DeepSeek-Coder 不仅在多个基准测试中在开源代码模型中取得了最先进的性能，而且还超越了现有的闭源模型，如 Codex 和 GPT-3.5。此外，DeepSeek-Coder 模型采用宽松许可证，允许研究和无限制的商业使用。

The rapid development of large language models has revolutionized code intelligence in software development. However, the predominance of closed-source models has restricted extensive research and development. To address this, we introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling. Our extensive evaluations demonstrate that DeepSeek-Coder not only achieves state-of-the-art performance among open-source code models across multiple benchmarks but also surpasses existing closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use.

核心贡献 · Key contributions

• 提出 DeepSeek-Coder 系列，开源代码模型，参数规模从 1.3B 到 33B。
  Introduces DeepSeek-Coder series, open-source code models from 1.3B to 33B parameters.
• 首次在预训练中引入仓库级数据构建，提升跨文件代码生成能力。
  First to incorporate repository-level data construction during pre-training, boosting cross-file code generation.
• 严格分析 FIM 训练策略，揭示权衡关系与最优配置。
  Rigorously examines FIM training strategies, revealing trade-offs and optimal configurations.
• 在多个基准测试中达到开源代码模型的最优性能。
  Achieves state-of-the-art performance among open-source code models across multiple benchmarks.
• 微调后的 DeepSeek-Coder-Instruct 33B 在代码任务上超越 GPT-3.5 Turbo。
  DeepSeek-Coder-Instruct 33B surpasses GPT-3.5 Turbo in code-related tasks after fine-tuning.
• 以宽松许可证发布模型，支持研究和商业用途。
  Releases models under permissive license for research and commercial use.

局限 · Limitations

• 评估限于 87 种编程语言，其他语言性能未经测试。
  Evaluation limited to 87 programming languages; performance on other languages untested.
• 长上下文适配仅测试至 16K token，超出范围的可靠性未验证。
  Long-context adaptation only tested up to 16K tokens; reliability beyond that unverified.
• 无法完全排除 GitHub 测试集的数据污染风险。
  Potential data contamination from GitHub test sets cannot be fully ruled out.
• FIM 训练在代码补全能力与填充性能之间存在权衡。
  FIM training trades off code completion ability for infilling performance.
• DeepSeek-Coder-v1.5 在 NLP 提升的同时编码性能略有下降。
  DeepSeek-Coder-v1.5 shows slight coding performance drop despite improved NLP.

论文章节 · Sections（共 22）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2 数据收集 2 Data Collection
• 2.1 GitHub 数据爬取与过滤 2.1 GitHub Data Crawling and Filtering
• 2.2 依赖解析 2.2 Dependency Parsing
• 2.3 仓库级去重 2.3 Repo-Level Deduplication
• 2.4 质量筛选与去污染 2.4 Quality Screening and Decontamination
• 3.1 训练策略 3.1 Training Strategy
• 3.2 分词器 3.2 Tokenizer
• 3.3 模型架构 3.3 Model Architecture
• 3.4 优化 3.4 Optimization
• 3.5 环境 3.5 Environments
• 3.6 长上下文 3.6 Long Context
• 3.7 指令微调 3.7 Instruction Tuning
• 4 实验结果 4 Experimental Results
• 4.1 代码生成 4.1 Code Generation
• 4.2 中间填充代码补全 4.2 Fill-in-the-Middle Code Completion
• 4.3 跨文件代码补全 4.3 Cross-File Code Completion
• 4.4 基于程序的数学推理 4.4 Program-based Math Reasoning
• 5 从通用大语言模型继续预训练 5 Continue Pre-Training From General LLM
• 6 结论 6 Conclusion
• 致谢 Acknowledgements

阅读逐段中英对照全文 →