Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU. We also publicly release Flan-T5 checkpoints, which achieve strong few-shot performance even compared to much larger models, such as PaLM 62B. Overall, instruction finetuning is a general method for improving the performance and usability of pretrained language models.
核心贡献 · Key contributions
将指令微调扩展到 540B 模型和 1.8K 任务，展示模型和任务规模扩张带来的性能提升。 Scales instruction finetuning to 540B models and 1.8K tasks, showing performance gains with model and task scaling.
证明在微调中加入思维链数据能提升在留出任务上的推理能力。 Demonstrates that including chain-of-thought data in finetuning improves reasoning on held-out tasks.
展示指令微调在不同架构（编码器-解码器、仅解码器）和模型规模上的泛化性。 Shows instruction finetuning generalizes across architectures (encoder-decoder, decoder-only) and model sizes.
在 MMLU 上达到最优（五次样本 75.2%），并提升 TyDiQA 等多语言基准。 Achieves state-of-the-art on MMLU (75.2% five-shot) and improves multilingual benchmarks like TyDiQA.
发布 Flan-T5 检查点，在少样本任务上超越 PaLM 62B 等更大模型。 Releases Flan-T5 checkpoints that outperform much larger models like PaLM 62B on few-shot tasks.
指令微调计算高效，Flan-PaLM 540B 仅用预训练算力的 0.2%。 Instruction finetuning is compute-efficient, using only 0.2% of pre-training compute for Flan-PaLM 540B.
局限 · Limitations
任务数量超过 282 个后规模扩张收益递减。 Scaling number of tasks shows diminishing returns beyond 282 tasks.
Flan-PaLM 在算法类 BBH 任务上未超越专用模型。 Flan-PaLM does not surpass specialized models on algorithmic BBH tasks.
不含思维链数据的指令微调会降低在思维链任务上的推理性能。 Instruction finetuning without CoT data degrades reasoning performance on CoT tasks.
评估限于英语中心和多语言基准，未在所有语言上测试。 Evaluation limited to English-centric and multilingual benchmarks; not tested on all languages.
人工评估每个示例仅由一名评分者完成，可能引入主观性。 Human evaluation uses single rater per example, which may introduce subjectivity.
论文章节 · Sections（共 11）
摘要Abstract
1 引言1 Introduction
2 Flan 微调2 Flan Finetuning
3 扩展到 540B 参数和 1.8K 任务3 Scaling to 540B parameters and 1.8K tasks
4 使用思维链注释进行微调4 Finetuning with chain-of-thought annotations
5 综合应用5 Putting it all together
6 开放式生成的可用性评估6 Usability evaluation of open-ended generation