ELECTRA：将文本编码器预训练为判别器而非生成器

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

凯文·克拉克 Kevin Clark · Stanford · 2020-03-23 · arXiv:2003.10555 ↗

摘要 · Abstract

掩码语言建模（MLM）预训练方法（如 BERT）通过用[MASK]替换部分令牌来破坏输入，然后训练模型重建原始令牌。虽然这些方法在迁移到下游 NLP 任务时表现良好，但通常需要大量计算才能有效。作为替代，我们提出了一种更高效的预训练任务，称为替换令牌检测。我们的方法不是掩码输入，而是通过从小型生成器网络中采样合理的替代令牌来替换部分令牌，从而破坏输入。然后，我们不训练模型预测被破坏令牌的原始身份，而是训练一个判别模型，预测被破坏输入中的每个令牌是否被生成器样本替换。大量实验表明，这种新的预训练任务比 MLM 更高效，因为该任务定义在所有输入令牌上，而不仅仅是掩码掉的一小部分。因此，在相同模型大小、数据和计算量下，我们的方法学习到的上下文表示显著优于 BERT。对于小模型，增益尤为显著；例如，我们在一个 GPU 上训练 4 天的模型，在 GLUE 自然语言理解基准测试中超越了使用 30 倍计算量训练的 GPT。我们的方法在大规模下也表现良好，使用不到 1/4 的计算量即可与 RoBERTa 和 XLNet 相媲美，并在相同计算量下超越它们。

Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.

核心贡献 · Key contributions

提出替换词检测，一种比掩码语言建模更高效的预训练任务。
Proposes replaced token detection, a more sample-efficient pre-training task than masked language modeling.
对所有输入词元训练判别模型，而非仅掩码子集，提升计算效率。
Trains a discriminative model on all input tokens, not just a masked subset, improving compute efficiency.
证明 ELECTRA 在相同算力或模型规模下优于 BERT、GPT、RoBERTa 和 XLNet。
Demonstrates ELECTRA outperforms BERT, GPT, RoBERTa, and XLNet under same compute or model size.
小模型表现突出，如 ELECTRA-Small 在 GLUE 上以 30 倍更少算力超越 GPT。
Shows strong gains for small models, e.g., ELECTRA-Small beats GPT on GLUE with 30x less compute.
ELECTRA-Large 在 SQuAD 2.0 上达到最优，且算力少于先前方法。
Achieves state-of-the-art on SQuAD 2.0 with ELECTRA-Large while using less compute than prior methods.
引入权重共享和更小生成器以进一步提升效率和性能。
Introduces weight sharing and smaller generators to further improve efficiency and performance.

局限 · Limitations

评估仅限于英文文本，未探索多语言适用性。
Evaluation limited to English text; multilingual applicability not explored.
生成器使用最大似然训练而非对抗训练，限制了潜在收益。
Generator trained with maximum likelihood, not adversarially, limiting potential gains.
两阶段训练和对抗训练不如联合训练，改进有限。
Two-stage training and adversarial training underperform joint training, showing limited improvement.
大规模对比因训练数据和超参数不同而不完全公平。
Large-scale comparisons not fully apples-to-apples due to different training data and hyperparameters.
参数效率未完全解释，需更多分析以全面理解。
Parameter efficiency not fully explained; more analysis needed for complete understanding.

论文章节 · Sections（共 11）

摘要 Abstract
1 引言 1 Introduction
2 方法 2 Method
3.1 实验设置 3.1 Experimental Setup
3.2 模型扩展 3.2 Model Extensions
3.3 小模型 3.3 Small Models
3.4 大模型 3.4 Large Models
3.5 效率分析 3.5 Efficiency Analysis
4 相关工作 4 Related Work
5 结论 5 Conclusion
致谢 Acknowledgements

阅读逐段中英对照全文 →