We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.
核心贡献 · Key contributions
提出 Codex，一个在 GitHub 代码上微调的 GPT 模型，在 HumanEval 上达到 28.8%的 pass@1。 Introduces Codex, a GPT model fine-tuned on GitHub code, achieving 28.8% pass@1 on HumanEval.
展示重复采样（100 个样本）可解决 70.2%的 HumanEval 问题。 Demonstrates repeated sampling with 100 samples solves 70.2% of HumanEval problems.
发布包含 164 个手写编程问题和单元测试的 HumanEval 数据集。 Releases HumanEval dataset of 164 hand-written programming problems with unit tests.
展示监督微调（Codex-S）使 pass@1 提升 6.5 个百分点，pass@100 提升 15.1 个百分点。 Shows supervised fine-tuning (Codex-S) improves pass@1 by 6.5 points and pass@100 by 15.1 points.
发现 BLEU 分数对功能正确性不可靠，功能正确性更优。 Finds BLEU score unreliable for functional correctness; functional correctness is preferred.
讨论更广泛的影响，包括安全、安保、经济和一致性问题。 Discusses broader impacts including safety, security, economics, and alignment issues.
局限 · Limitations
Codex 难以处理文档字符串中的长操作链以及将操作绑定到变量。 Codex struggles with long chains of operations in docstrings and binding operations to variables.
Codex 样本效率不高，相比人类程序员需要大量训练数据。 Codex is not sample efficient; requires huge training data compared to human programmers.
Codex 可能生成语法错误、未定义或不安全的代码，并与用户意图不一致。 Codex can generate syntactically incorrect, undefined, or insecure code, and may be misaligned with user intent.
在合成任务中，Codex 性能随文档字符串长度增加呈指数下降。 Codex performance degrades exponentially with increasing docstring length in synthetic tasks.
Codex 可能生成带有偏见或有害的代码注释，反映刻板印象。 Codex may produce biased or harmful outputs as code comments, reflecting stereotypes.
论文章节 · Sections（共 32）
摘要Abstract
1 引言1 Introduction
2 评估框架2 Evaluation Framework
2.1 功能正确性2.1 Functional Correctness
2.2 HumanEval：手工编写的评估集2.2 HumanEval: Hand-Written Evaluation Set
2.3 执行生成程序的沙箱2.3 Sandbox for Executing Generated Programs
3 代码微调3 Code Fine-Tuning
3.1 数据收集3.1 Data Collection
3.2 方法3.2 Methods
3.3 结果3.3 Results
3.4 相关模型与系统的对比分析3.4 Comparative Analysis of Related Models and Systems
3.5 在 APPS 数据集上的结果3.5 Results on the APPS Dataset
4 监督微调4 Supervised Fine-Tuning
4.1 竞赛编程问题4.1 Problems from Competitive Programming
4.2 持续集成问题4.2 Problems from Continuous Integration
4.3 问题筛选4.3 Filtering Problems
4.4 方法4.4 Methods
4.5 结果4.5 Results
5 文档字符串生成5 Docstring Generation
6 局限性6 Limitations
7 更广泛的影响与危害分析7 Broader Impacts and Hazard Analysis
7.1 过度依赖7.1 Over-reliance
7.2 错位7.2 Misalignment
7.3 偏见与代表性7.3 Bias and representation
7.4 经济与劳动力市场影响7.4 Economic and labor market impacts