Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.
核心贡献 · Key contributions
证明 BERT 训练不足，提出 RoBERTa 改进训练方案。 Shows BERT was significantly undertrained and proposes RoBERTa with improved training recipe.
证明去除下一句预测目标可匹配或提升性能。 Demonstrates that removing the next sentence prediction objective matches or improves performance.
引入动态掩码、大批量和字节级 BPE 以改进预训练。 Introduces dynamic masking, large mini-batches, and byte-level BPE for better pretraining.
收集 CC-News 数据集，表明更多数据提升下游任务性能。 Collects CC-News dataset and shows more data improves downstream task performance.
在 GLUE、RACE 和 SQuAD 上取得最优结果，无需多任务微调。 Achieves state-of-the-art results on GLUE, RACE, and SQuAD without multi-task finetuning.
发布预训练和微调的模型与代码。 Releases models and code for pretraining and finetuning.
局限 · Limitations
研究聚焦 BERT 架构，未探索架构变化。 Study focuses on BERT architecture; architectural changes not explored.
结果可能不泛化到其他预训练目标或模型规模。 Results may not generalize to other pretraining objectives or model sizes.
数据规模与多样性效应混淆，未分离分析。 Data size and diversity effects are conflated; not disentangled.
大批量训练极限未充分探索，进一步扩展可能。 Large batch training limits not fully explored; further scaling possible.
与其他方法的比较可能受超参数调优差异影响。 Comparison with other methods may be affected by hyperparameter tuning differences.
论文章节 · Sections（共 23）
摘要Abstract
1 引言1 Introduction
2 背景2 Background
2.1 设置2.1 Setup
2.2 架构2.2 Architecture
2.3 训练目标2.3 Training Objectives
2.4 优化2.4 Optimization
2.5 数据2.5 Data
3 实验设置3 Experimental Setup
3.1 实现3.1 Implementation
3.2 数据3.2 Data
3.3 评估3.3 Evaluation
4 训练过程分析4 Training Procedure Analysis
4.1 静态与动态掩码4.1 Static vs. Dynamic Masking
4.2 模型输入格式与下一句预测4.2 Model Input Format and Next Sentence Prediction