We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.
核心贡献 · Key contributions
发布基于 Llama 2 的代码大语言模型系列 Code Llama，在开放模型中达到最先进性能。 Releases Code Llama, a family of large language models for code based on Llama 2 with state-of-the-art performance among open models.
提供多种变体：基础模型（Code Llama）、Python 专用模型（Code Llama - Python）和指令遵循模型（Code Llama - Instruct），参数规模为 7B、13B、34B 和 70B。 Provides multiple variants: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) in 7B, 13B, 34B, and 70B sizes.
支持 7B、13B 和 70B 模型的代码填充（中间填充）功能，可用于 IDE 中的代码补全。 Supports infilling (fill-in-the-middle) for 7B, 13B, and 70B models, enabling code completion in IDEs.
通过修改 RoPE 基频的长上下文微调（LCFT），将上下文长度扩展到 100K 词元。 Extends context length to 100K tokens via long context fine-tuning (LCFT) with modified RoPE base period.
在 HumanEval（67% pass@1）和 MBPP（65% pass@1）上达到开放模型中的最先进结果。 Achieves state-of-the-art results on HumanEval (67% pass@1) and MBPP (65% pass@1) among open models.
Code Llama - Python 7B 在 HumanEval 和 MBPP 上超越 Llama 2 70B，所有模型在 MultiPL-E 上超越其他公开模型。 Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all models surpass other public models on MultiPL-E.
局限 · Limitations
填充和长上下文微调在标准短序列基准（如 HumanEval 和 MBPP）上略微降低性能。 Infilling and long context fine-tuning slightly degrade performance on standard short-sequence benchmarks like HumanEval and MBPP.
34B 模型缺乏填充能力，限制了其在实时代码补全场景中的应用。 The 34B model lacks infilling capability, limiting its use in real-time code completion scenarios.
长上下文微调在短提示（低于 4K 词元）上性能下降最多 2 个 BLEU 点。 Long context fine-tuning reduces performance on short prompts (under 4K tokens) by up to 2 BLEU points.
安全评估依赖自动基准和有限的红队测试，可能无法覆盖所有风险。 Safety evaluations rely on automatic benchmarks and limited red teaming, which may not capture all risks.
指令微调可能导致对良性提示的过度拒绝或错误拒绝，红队测试已指出此问题。 Instruction fine-tuning may cause over-refusal or false refusals on benign prompts, as noted in red teaming.
论文章节 · Sections（共 16）
摘要Abstract
1 引言1 Introduction
2.1 Code Llama 模型系列2.1 The Code Llama models family