GLU 变体改进 Transformer

GLU Variants Improve Transformer

诺姆·沙泽尔 Noam Shazeer · Google · 2020-02-12 · arXiv:2002.05202 ↗ · 被引 2011

摘要 · Abstract

门控线性单元（GLU，arXiv:1612.08083）由两个线性投影的分量积组成，其中一个先经过 sigmoid 函数。GLU 可以有多种变体，使用不同的非线性（甚至线性）函数代替 sigmoid。我们在 Transformer（arXiv:1706.03762）序列到序列模型的前馈子层中测试了这些变体，发现其中一些变体比通常使用的 ReLU 或 GELU 激活函数能带来质量提升。

Gated Linear Units (arXiv:1612.08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers of the Transformer (arXiv:1706.03762) sequence-to-sequence model, and find that some of them yield quality improvements over the typically-used ReLU or GELU activations.

核心贡献 · Key contributions

提出将 GLU 变体（GEGLU、SwiGLU）用于 Transformer 前馈网络层。
Proposes GLU variants (GEGLU, SwiGLU) for Transformer FFN layers.
展示 GLU 变体在去噪预训练目标上提升了困惑度。
Shows GLU variants improve perplexity on denoising pre-training objective.
证明在 GLUE 和 SuperGLUE 下游任务上性能更优。
Demonstrates better downstream task performance on GLUE and SuperGLUE.
通过将三矩阵层的隐藏大小减少 2/3 来保持参数量不变。
Maintains parameter count by reducing hidden size by 2/3 for three-matrix layers.
发现预训练期间不使用 dropout 可获得更优结果。
Finds no dropout during pre-training yields superior results.
提供简单实现且无计算缺陷。
Provides simple implementation with no computational drawbacks.

局限 · Limitations

仅在编码器-解码器 Transformer 和 T5 式预训练上测试。
Only tested on encoder-decoder Transformer with T5-style pre-training.
未提供 GLU 变体提升性能的理论解释。
No theoretical explanation for why GLU variants improve performance.
微调在单一任务混合上进行，而非每个任务单独微调。
Fine-tuning done on a single mixture of tasks, not per-task.
结果可能无法泛化到其他架构或训练设置。
Results may not generalize to other architectures or training setups.
仅限于前馈子层；Transformer 其他组件未变。
Limited to feed-forward sublayers; other Transformer components unchanged.

论文章节 · Sections（共 8）

摘要 Abstract
1 引言 1 Introduction
2 门控线性单元（GLU）及其变体 2 Gated Linear Units (GLU) and Variants
3 文本到文本迁移变换器（T5）的实验 3 Experiments on Text-to-Text Transfer Transformer (T5)
3.1 模型架构 3.1 Model Architecture
3.2 预训练与困惑度结果 3.2 Pre-Training and Perplexity Results
3.3 微调 3.3 Fine-Tuning
4 结论 4 Conclusions

阅读逐段中英对照全文 →