Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at https://github.com/google-research/ALBERT.
核心贡献 · Key contributions
提出两种参数缩减技术：分解式嵌入参数化和跨层参数共享。 Proposes two parameter-reduction techniques: factorized embedding parameterization and cross-layer parameter sharing.
引入用于句子顺序预测（SOP）的自监督损失，以建模句子间连贯性。 Introduces a self-supervised loss for sentence-order prediction (SOP) to model inter-sentence coherence.
在 GLUE、SQuAD 和 RACE 基准上取得最先进结果，且参数少于 BERT-large。 Achieves state-of-the-art results on GLUE, SQuAD, and RACE benchmarks with fewer parameters than BERT-large.
证明参数共享起到正则化作用，稳定训练并提升泛化能力。 Demonstrates that parameter sharing acts as regularization, stabilizing training and improving generalization.
表明 dropout 可能损害大型 Transformer 模型的性能，因此移除它以获得更好结果。 Shows that dropout can hurt performance in large Transformer-based models, leading to its removal for better results.
局限 · Limitations
ALBERT-xxlarge 由于结构更大，计算成本高于 BERT-large。 ALBERT-xxlarge is computationally more expensive than BERT-large due to larger structure.
研究仅限于英语和特定基准（GLUE、SQuAD、RACE）。 The study is limited to English language and specific benchmarks (GLUE, SQuAD, RACE).
提出的 SOP 损失可能无法捕捉语篇连贯性的所有维度。 The proposed SOP loss may not capture all dimensions of discourse coherence.
移除 dropout 的效果可能不适用于其他 Transformer 架构。 The effect of removing dropout may not generalize to other Transformer architectures.
论文章节 · Sections（共 18）
摘要Abstract
1 引言1 Introduction
2.1 自然语言表示学习的规模化2.1 Scaling Up Representation Learning for Natural Language
2.2 跨层参数共享2.2 Cross-layer parameter sharing
2.3 句子排序目标2.3 Sentence Ordering Objectives
3 ALBERT 的要素3 The Elements of ALBERT
3.1 模型架构选择3.1 Model architecture choices
3.2 模型设置3.2 Model setup
4.1 实验设置4.1 Experimental Setup
4.2 评估基准4.2 Evaluation Benchmarks
4.3 BERT 与 ALBERT 的整体比较4.3 Overall Comparison between BERT and ALBERT
4.5 跨层参数共享4.5 Cross-layer parameter sharing
4.6 句子顺序预测（SOP）4.6 Sentence order prediction (SOP)
4.7 如果训练时间相同会怎样？4.7 What if we train for the same amount of time?
4.8 额外训练数据和 dropout 效果4.8 Additional training data and dropout effects
4.9 NLU 任务当前最优结果4.9 Current State-of-the-art on NLU Tasks