Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A significant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies.
核心贡献 · Key contributions
使用 Pathways 在 6144 个 TPU v4 芯片上训练了密集激活的 Transformer 模型 PaLM 540B，实现了高效率。 Trained PaLM 540B, a densely activated Transformer, using Pathways across 6144 TPU v4 chips, achieving high efficiency.
在数百个语言理解和生成基准上取得了最先进的少样本学习结果。 Achieved state-of-the-art few-shot results on hundreds of language understanding and generation benchmarks.
使用思维链提示在多步推理任务上展示了突破性性能，超越了微调模型。 Demonstrated breakthrough performance on multi-step reasoning tasks using chain-of-thought prompting, outperforming finetuned models.
在许多 BIG-bench 任务上观察到模型规模带来的不连续改进，表明出现了新能力。 Observed discontinuous improvements from model scale on many BIG-bench tasks, indicating emergent capabilities.
展示了强大的多语言和代码生成能力，PaLM 540B 缩小了与微调模型的差距。 Showed strong multilingual and code generation capabilities, with PaLM 540B bridging the gap to finetuned models.
提供了对偏见、毒性和记忆化的全面分析，并讨论了伦理考量。 Provided comprehensive analysis of bias, toxicity, and memorization, and discussed ethical considerations.
局限 · Limitations
最大模型在训练中出现损失尖峰，通过从较早检查点重启来缓解。 Training loss spikes occurred for the largest model, mitigated by restarting from earlier checkpoints.
评估仅限于英语和少数多语言任务；非英语数据比例仅为 22%。 Evaluation limited to English and a few multilingual tasks; non-English data proportion was only 22%.
偏见和毒性分析显示较大模型存在刻板印象和更高毒性，与提示风格相关。 Bias and toxicity analysis showed stereotypes and higher toxicity for larger models, correlating with prompt style.
思维链提示需要手动示例；在某些 BIG-bench 任务上性能仍远低于人类。 Chain-of-thought prompting required manual exemplars; performance on some BIG-bench tasks still far from human.
代码生成性能依赖于额外微调；仅模型规模并未使改进饱和。 Code generation performance relied on additional finetuning; model size alone did not saturate improvements.
论文章节 · Sections（共 30）
摘要Abstract
1 引言1 Introduction
2 模型架构2 Model Architecture
2.1 模型规模超参数2.1 Model Scale Hyperparameters
2.2 模型卡2.2 Model Card
3 训练数据集3 Training Dataset
4 训练基础设施4 Training Infrastructure
4.1 训练效率4.1 Training Efficiency
5 训练设置5 Training Setup
5.1 训练不稳定性5.1 Training Instability
6.1 英文 NLP 任务6.1 English NLP tasks
6.2 BIG-bench6.2 BIG-bench
6.3 推理6.3 Reasoning
6.4 代码任务6.4 Code Tasks
6.5 翻译6.5 Translation
6.6 多语言自然语言生成6.6 Multilingual Natural Language Generation
6.7 多语言问答6.7 Multilingual Question Answering
6.8 分析6.8 Analysis
7 记忆化7 Memorization
8 数据集污染8 Dataset Contamination
9 探索解释9 Exploring Explanations
10 表征偏差分析10 Representational Bias Analysis
10.1 社会群体中的分布偏差10.1 Distributional bias in social groups
10.2 开放生成中的毒性10.2 Toxicity in open-ended generation