Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch.
核心贡献 · Key contributions
提出带有段级循环机制的 Transformer-XL，用于学习长期依赖。 Proposes Transformer-XL with segment-level recurrence for long-term dependency learning.
引入新型相对位置编码方案，支持状态复用。 Introduces a novel relative positional encoding scheme enabling state reuse.
在五个语言建模基准上取得最优结果。 Achieves state-of-the-art results on five language modeling benchmarks.
解决上下文碎片问题，提升短序列性能。 Resolves context fragmentation problem, improving performance on short sequences.
推理速度比原始 Transformer 快 1800 倍以上。 Demonstrates up to 1,800+ times faster evaluation than vanilla Transformers.
生成数千词元的连贯长文本文章。 Generates coherent long text articles with thousands of tokens.
局限 · Limitations
循环机制需要额外内存来缓存隐藏状态。 Recurrence mechanism requires additional memory for caching hidden states.
训练仍使用固定长度段，限制理论上下文长度。 Training still uses fixed-length segments, limiting theoretical context length.
相对位置编码可能无法泛化到极长序列。 Relative positional encoding may not generalize to extremely long sequences.
在小数据集上的性能依赖正则化技术。 Performance on small datasets relies on regularization techniques.
当记忆规模极大时，推理速度优势减弱。 Evaluation speed advantage diminishes with very large memory sizes.
论文章节 · Sections（共 13）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 模型3 Model
3.1 基础 Transformer 语言模型3.1 Vanilla Transformer Language Models
3.2 基于状态复用的段级循环3.2 Segment-Level Recurrence with State Reuse