BERT: 深度双向 Transformer 预训练语言理解模型

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

雅各布·德夫林 Jacob Devlin · Google · 2018-10-11 · arXiv:1810.04805 ↗ · 被引 117033

摘要 · Abstract

我们提出了一种名为 BERT 的新语言表示模型，它代表来自 Transformer 的双向编码器表示。与最近的语言表示模型不同，BERT 旨在通过在所有层中联合调节左右上下文，从未标注文本中预训练深度双向表示。因此，预训练的 BERT 模型只需添加一个额外的输出层进行微调，就能在广泛的任务（如问答和语言推理）中创建最先进的模型，而无需大量特定于任务的架构修改。BERT 概念简单且经验上强大。它在 11 项自然语言处理任务上取得了新的最先进结果，包括将 GLUE 分数提升至 80.5%（绝对提升 7.7 个百分点）、MultiNLI 准确率提升至 86.7%（绝对提升 4.6 个百分点）、SQuAD v1.1 问答测试 F1 提升至 93.2（绝对提升 1.5 个百分点）以及 SQuAD v2.0 测试 F1 提升至 83.1（绝对提升 5.1 个百分点）。

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

核心贡献 · Key contributions

提出 BERT，一种通过掩码语言建模和下一句预测预训练的深度双向 Transformer。
Introduces BERT, a deep bidirectional Transformer pre-trained via masked language modeling and next sentence prediction.
证明双向预训练在句子级和词元级任务上显著优于单向方法。
Demonstrates that bidirectional pre-training significantly outperforms unidirectional approaches on sentence-level and token-level tasks.
在 GLUE、SQuAD 和 SWAG 等 11 项 NLP 任务上取得大幅领先的最优结果。
Achieves state-of-the-art results on 11 NLP tasks, including GLUE, SQuAD, and SWAG, with substantial margins.
表明预训练表示减少了对任务特定架构的需求，支持简单微调。
Shows that pre-trained representations reduce the need for task-specific architectures, enabling simple fine-tuning.
提供证据表明，预训练后扩大模型规模即使在小型数据集上也能提升性能。
Provides evidence that scaling model size improves performance even on small datasets when pre-trained.
验证了 BERT 在下游任务中微调和基于特征的方法均有效。
Validates both fine-tuning and feature-based approaches for downstream tasks using BERT.

局限 · Limitations

预训练需要大量算力和大规模文本语料。
Pre-training requires large computational resources and extensive text corpora.
掩码语言建模因[MASK]词元导致预训练与微调之间存在不一致。
Masked language modeling creates a mismatch between pre-training and fine-tuning due to [MASK] tokens.
下一句预测任务可能无法很好地泛化到所有下游任务。
The next sentence prediction task may not generalize well to all downstream tasks.
微调在小型数据集上可能不稳定，需要多次随机重启。
Fine-tuning can be unstable on small datasets, requiring multiple random restarts.
模型性能仅针对英语任务和数据集进行了评估。
The model's performance is evaluated only on English-language tasks and datasets.

论文章节 · Sections（共 19）

摘要 Abstract
1 引言 1 Introduction
2 相关工作 2 Related Work
2.1 无监督基于特征的方法 2.1 Unsupervised Feature-based Approaches
2.2 无监督微调方法 2.2 Unsupervised Fine-tuning Approaches
2.3 从监督数据迁移学习 2.3 Transfer Learning from Supervised Data
3 BERT 3 BERT
3.1 预训练 BERT 3.1 Pre-training BERT
3.2 微调 BERT 3.2 Fine-tuning BERT
4 实验 4 Experiments
4.1 GLUE 4.1 GLUE
4.2 SQuAD v1.1 4.2 SQuAD v1.1
4.3 SQuAD v2.0 4.3 SQuAD v2.0
4.4 SWAG 4.4 SWAG
5 消融研究 5 Ablation Studies
5.1 预训练任务的影响 5.1 Effect of Pre-training Tasks
5.2 模型大小的影响 5.2 Effect of Model Size
5.3 基于特征的 BERT 方法 5.3 Feature-based Approach with BERT
6 结论 6 Conclusion

阅读逐段中英对照全文 →