Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.
核心贡献 · Key contributions
证明规模扩张语言模型大幅提升任务无关的少样本性能，有时可媲美微调模型。 Demonstrates that scaling up language models greatly improves task-agnostic few-shot performance, sometimes matching fine-tuned models.
训练了 1750 亿参数的自回归模型 GPT-3，并在多种 NLP 任务上测试其少样本、单样本和零样本性能。 Trains GPT-3, a 175 billion parameter autoregressive model, and tests its few-shot, one-shot, and zero-shot performance on diverse NLP tasks.
在翻译、问答、完形填空以及即时推理任务（如算术和单词重组）上取得强劲结果。 Achieves strong results on translation, question-answering, cloze tasks, and on-the-fly reasoning tasks like arithmetic and word unscrambling.
识别出少样本学习仍困难的领域，如自然语言推理和部分阅读理解数据集。 Identifies tasks where few-shot learning struggles, such as natural language inference and some reading comprehension datasets.
发现 GPT-3 生成的新闻文章人类评估者难以区分与人类撰写的文章。 Finds that GPT-3 can generate news articles indistinguishable from human-written ones by human evaluators.
显示大多数任务中性能随模型规模平滑提升，更大模型从上下文学习中获益更多。 Shows smooth scaling of performance with model size across most tasks, with larger models benefiting more from in-context learning.
局限 · Limitations
即使在 GPT-3 上，ANLI 等自然语言推理任务的少样本性能仍接近随机水平。 Few-shot performance on natural language inference tasks like ANLI remains near random chance even for GPT-3.
GPT-3 在需要比较两个句子的任务上表现不佳，如 SuperGLUE 中的 WiC、RTE 和 CB。 GPT-3 struggles with tasks requiring comparing two sentences, such as WiC, RTE, and CB in SuperGLUE.
在网页语料上训练可能导致数据污染，在某些数据集上夸大结果，但总体影响较小。 Data contamination from training on web corpora may inflate results on some datasets, though impact is minimal overall.
GPT-3 在 RACE 和 QuAC 等阅读理解数据集上表现较弱，远落后于最先进的微调模型。 GPT-3's performance on reading comprehension datasets like RACE and QuAC is weak, far behind state-of-the-art fine-tuned models.
模型训练和推理需要大量算力，限制了可访问性和可复现性。 The model requires large computational resources for training and inference, limiting accessibility and reproducibility.
论文章节 · Sections（共 35）
摘要Abstract
1 引言1 Introduction
2 方法2 Approach
2.1 模型与架构2.1 Model and Architectures
2.2 训练数据集2.2 Training Dataset
2.3 训练过程2.3 Training Process
2.4 评估2.4 Evaluation
3 结果3 Results
3.1 语言建模、完形填空与补全任务3.1 Language Modeling, Cloze, and Completion Tasks
3.2 闭卷问答3.2 Closed Book Question Answering
3.3 翻译3.3 Translation
3.4 威诺格拉德风格任务3.4 Winograd-Style Tasks
3.5 常识推理3.5 Common Sense Reasoning
3.6 阅读理解3.6 Reading Comprehension
3.7 SuperGLUE3.7 SuperGLUE
3.8 自然语言推理3.8 NLI
3.9 合成与定性任务3.9 Synthetic and Qualitative Tasks
4 衡量与防止基准记忆4 Measuring and Preventing Memorization Of Benchmarks
5 局限性5 Limitations
6 广泛影响6 Broader Impacts
6.1 语言模型的滥用6.1 Misuse of Language Models
6.2 公平性、偏见与代表性6.2 Fairness, Bias, and Representation
6.3 能源消耗6.3 Energy Usage
7 相关工作7 Related Work
8 结论8 Conclusion
致谢Acknowledgements
贡献Contributions
附录 A 通用爬虫过滤细节Appendix A Details of Common Crawl Filtering
附录 B 模型训练细节Appendix B Details of Model Training
附录 C 测试集污染研究细节Appendix C Details of Test Set Contamination Studies
附录 D 训练语言模型的总计算量Appendix D Total Compute Used to Train Language Models
附录 E 合成新闻文章的人类质量评估Appendix E Human Quality Assessment of Synthetic News Articles
附录 F GPT-3 的额外样本Appendix F Additional Samples from GPT-3
附录 G 任务表述与规范细节Appendix G Details of Task Phrasing and Specifications
附录 H 所有模型规模在所有任务上的结果Appendix H Results on All Tasks for All Model Sizes