Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models.
核心贡献 · Key contributions
将涌现能力定义为大模型具备而小模型不具备的能力，无法通过缩放定律预测。 Defines emergent abilities as abilities present in large models but absent in smaller ones, unpredictable via scaling laws.
记录了少样本提示、思维链、指令遵循和程序执行等场景中的涌现能力。 Documents emergent abilities across few-shot prompting, chain-of-thought, instruction following, and program execution.
展示涌现能力在缩放曲线中表现为相变，性能在阈值处从随机跃升至显著高于随机。 Shows emergent abilities appear as phase transitions in scaling curves, with performance jumping from random to above-random at a threshold.
讨论涌现能力并非仅由规模决定；架构、数据质量和训练过程同样重要。 Discusses that emergent abilities are not solely determined by scale; architecture, data quality, and training procedures also matter.
强调随规模增加而涌现的风险，如偏见、毒性和记忆化。 Highlights emergent risks such as bias, toxicity, and memorization that can increase with scale.
提出未来方向，包括进一步扩展、改进架构、数据扩展和理解涌现。 Proposes future directions including further scaling, improved architectures, data scaling, and understanding emergence.
局限 · Limitations
涌现能力仅针对预训练 Transformer 语言模型定义，不适用于其他架构。 Emergent abilities are defined only for pre-trained Transformer language models, not other architectures.
本文未提出新方法或模型，仅综述现有结果。 The paper does not propose new methods or models; it surveys existing results.
涌现可能是评估指标（如精确匹配）的假象，忽略了部分进展。 Emergence may be an artifact of evaluation metrics like exact match, which ignore partial progress.
涌现发生的规模并非固定，取决于数据质量、架构和训练。 The scale at which emergence occurs is not fixed; it depends on data quality, architecture, and training.
某些能力可能因分布不匹配或涌现后停滞而永远不会出现。 Some abilities may never emerge due to distributional mismatch or plateauing after emergence.
论文章节 · Sections（共 13）
摘要Abstract
1 引言1 Introduction
2 涌现能力定义2 Emergent Abilities Definition
3 少样本提示任务3 Few-Shot Prompted Tasks
4 增强提示策略4 Augmented Prompting Strategies
5 讨论5 Discussion
5.1 涌现的潜在解释5.1 Potential explanations of emergence