In this report, we introduce the Qwen2.5-Coder series, a significant upgrade from its predecessor, CodeQwen1.5. This series includes six models: Qwen2.5-Coder-(0.5B/1.5B/3B/7B/14B/32B). As a code-specific model, Qwen2.5-Coder is built upon the Qwen2.5 architecture and continues pretrained on a vast corpus of over 5.5 trillion tokens. Through meticulous data cleaning, scalable synthetic data generation, and balanced data mixing, Qwen2.5-Coder demonstrates impressive code generation capabilities while retaining general and math skills. These models have been evaluated on a wide range of code-related tasks, achieving state-of-the-art (SOTA) performance across more than 10 benchmarks, including code generation, completion, reasoning, and repair, consistently outperforming larger models of the same model size. We believe that the release of the Qwen2.5-Coder series will advance research in code intelligence and, with its permissive licensing, support wider adoption by developers in real-world applications.
核心贡献 · Key contributions
引入 Qwen2.5-Coder 系列，基于 Qwen2.5 架构并在 5.5 万亿 tokens 上预训练。 Introduced Qwen2.5-Coder series, built on Qwen2.5 with pretraining on 5.5 trillion tokens.
在包括 HumanEval 和 MBPP 在内的 10 多个代码基准上取得最先进性能。 Achieved state-of-the-art performance on over 10 code benchmarks, including HumanEval and MBPP.
开发了三阶段训练流程：文件级预训练、仓库级预训练和指令微调。 Developed a three-stage training pipeline: file-level pretraining, repo-level pretraining, and instruction tuning.
设计了多语言多智能体框架用于合成多样化指令数据。 Designed a multilingual multi-agent framework for synthesizing diverse instruction data.
使用 10-gram 重叠方法实现去污染以防止测试集泄露。 Implemented decontamination to prevent test set leakage using 10-gram overlap.
通过仓库级预训练展示了高达 128K tokens 的强长上下文能力。 Demonstrated strong long-context capability up to 128K tokens via repo-level pretraining.
局限 · Limitations
评估仅限于 1.5B 和 7B 参数模型，未探索更大规模。 Evaluation limited to 1.5B and 7B parameter models; larger sizes not explored.
长上下文评估仅在合成 Needle in the Code 任务上进行，未涉及真实仓库。 Long-context evaluation only on synthetic Needle in the Code task, not real-world repos.
数据混合比例（70%代码、20%文本、10%数学）可能无法泛化到其他领域。 Data mixing ratio (70% code, 20% text, 10% math) may not generalize to other domains.
指令微调数据质量依赖基于 LLM 的评分，可能引入偏差。 Instruction tuning data quality relies on LLM-based scoring, which may introduce bias.
多语言评估仅覆盖 8 种语言，长尾语言可能代表性不足。 Multilingual evaluation covers only 8 languages; long-tail languages may be underrepresented.
论文章节 · Sections（共 22）
摘要Abstract
1 引言1 Introduction
2 模型架构2 Model Architecture
3.1 预训练数据3.1 Pretraining Data
3.2 训练策略3.2 Training Policy
4.1 指令数据配方4.1 A Recipe for Instruction Data
4.2 训练策略4.2 Training Policy
5 去污染5 Decontamination
6 基座模型评估6 Evaluation on Base Models
6.1 代码生成6.1 Code Generation
6.2 代码补全6.2 Code Completion
6.3 代码推理6.3 Code Reasoning
6.4 数学推理6.4 Math Reasoning
6.5 通用自然语言6.5 General Natural Language
6.6 长上下文评估6.6 Long-Context Evaluation
7 指令模型评估7 Evaluation on Instruct Models
7.1 代码生成7.1 Code Generation
7.2 代码推理7.2 Code Reasoning
7.3 代码编辑7.3 Code Editing
7.4 文本到 SQL7.4 Text-to-SQL
7.5 数学推理与通用自然语言7.5 Math Reasoning and General Natural Language