探索迁移学习的极限：基于统一的文本到文本转换器

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

科林·拉费尔 Colin Raffel · Google · 2019-10-23 · arXiv:1910.10683 ↗ · 被引 26385

摘要 · Abstract

迁移学习，即模型先在数据丰富的任务上进行预训练，然后在下游任务上进行微调，已成为自然语言处理（NLP）中的一项强大技术。迁移学习的有效性催生了多种方法、方法论和实践。在本文中，我们通过引入一个统一的框架，将所有基于文本的语言问题转换为文本到文本格式，探索了 NLP 迁移学习技术的全貌。我们的系统研究比较了预训练目标、架构、无标签数据集、迁移方法以及其他因素在数十个语言理解任务上的表现。通过将我们的探索见解与规模以及我们新的“巨型清洁爬取语料库”相结合，我们在涵盖摘要、问答、文本分类等多个基准上取得了最先进的结果。为了促进未来 NLP 迁移学习的研究，我们发布了我们的数据集、预训练模型和代码。

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

核心贡献 · Key contributions

提出了一个统一的文本到文本框架，用于自然语言处理的迁移学习。
Proposes a unified text-to-text framework for NLP transfer learning.
系统比较了预训练目标、架构和数据集。
Systematically compares pre-training objectives, architectures, and data sets.
引入了用于预训练的巨型干净爬取语料库（C4）。
Introduces the Colossal Clean Crawled Corpus (C4) for pre-training.
通过规模扩张在多个自然语言处理基准上取得了最先进的结果。
Achieves state-of-the-art results on multiple NLP benchmarks via scaling.
发布预训练模型、代码和数据以促进未来研究。
Releases pre-trained models, code, and data to facilitate future research.
探索了包括编码器-解码器和前缀语言模型在内的架构变体。
Explores architectural variants including encoder-decoder and prefix LM.

局限 · Limitations

研究仅限于英语任务和基准。
Study limited to English-language tasks and benchmarks.
预训练仅使用英语数据；多语言任务需要额外数据。
Pre-training only on English data; multilingual tasks require additional data.
大规模实验的计算成本很高。
Computational cost of large-scale experiments is high.
由于成本原因，未对所有因素进行组合探索。
No combinatorial exploration of all factors due to cost.
结果可能无法推广到非文本模态或低资源语言。
Results may not generalize to non-text modalities or low-resource languages.

论文章节 · Sections（共 41）

摘要 Abstract
1 引言 1 Introduction
2 实验设置 2 Setup
2.1 模型 2.1 Model
2.2 巨型清洁爬取语料库 2.2 The Colossal Clean Crawled Corpus
2.3 下游任务 2.3 Downstream Tasks
2.4 输入与输出格式 2.4 Input and Output Format
3 实验 3 Experiments
3.1 基线 3.1 Baseline
3.2 架构 3.2 Architectures
3.3 无监督目标 3.3 Unsupervised Objectives
3.4 预训练数据集 3.4 Pre-training Data set
3.5 训练策略 3.5 Training Strategy
3.6 规模扩展 3.6 Scaling
3.7 综合考量 3.7 Putting It All Together
4 反思 4 Reflection
4.1 要点总结 4.1 Takeaways
4.2 展望 4.2 Outlook
A 贡献 A Contributions
B 将 WNLI 转换为文本到文本格式 B Converting WNLI to Our Text-to-Text Format
C CNN/Daily Mail 上的预测示例 C Example Predictions on CNN/Daily Mail
D 预处理示例 D Preprocessed Examples
D.1 CoLA D.1 CoLA
D.2 RTE D.2 RTE
D.3 MNLI D.3 MNLI
D.4 MRPC D.4 MRPC
D.5 QNLI D.5 QNLI
D.6 QQP D.6 QQP
D.7 SST2 D.7 SST2
D.8 STSB D.8 STSB
D.9 CB D.9 CB
D.10 COPA D.10 COPA
D.11 MultiRC D.11 MultiRC
D.12 WiC D.12 WiC
D.13 WSC 与 DPR D.13 WSC and DPR
D.14 CNN/Daily Mail D.14 CNN/Daily Mail
D.15 SQuAD D.15 SQuAD
D.16 WMT 英译德 D.16 WMT English to German
D.17 WMT 英译法 D.17 WMT English to French
D.18 WMT 英译罗 D.18 WMT English to Romanian
E 所有实验的每项任务得分 E Scores on Every Task for All Experiments

阅读逐段中英对照全文 →