Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We also introduce a) a variation on architecture and initialization to train deeper networks, b) the recomputation of attention matrices to save memory, and c) fast attention kernels for training. We call networks with these changes Sparse Transformers, and show they can model sequences tens of thousands of timesteps long using hundreds of layers. We use the same architecture to model images, audio, and text from raw bytes, setting a new state of the art for density modeling of Enwik8, CIFAR-10, and ImageNet-64. We generate unconditional samples that demonstrate global coherence and great diversity, and show it is possible in principle to use self-attention to model sequences of length one million or more.
核心贡献 · Key contributions
提出了注意力矩阵的稀疏分解，将复杂度降至 O(n√n)。 Introduced sparse factorizations of the attention matrix reducing complexity to O(n√n).
提出了架构和初始化改进以训练更深网络。 Proposed architectural and initialization changes to train deeper networks.
引入注意力矩阵重计算以节省训练内存。 Introduced recomputation of attention matrices to save memory during training.
开发了用于稀疏注意力操作的快速 GPU 内核。 Developed fast GPU kernels for sparse attention operations.
在 Enwik8、CIFAR-10 和 ImageNet-64 上实现了最先进的密度建模。 Achieved state-of-the-art density modeling on Enwik8, CIFAR-10, and ImageNet-64.
证明了自注意力可建模长度超过一百万的序列。 Demonstrated self-attention can model sequences of length one million or more.
局限 · Limitations
稀疏模式可能无法捕获所有数据依赖的注意力模式。 Sparse patterns may not capture all data-dependent attention patterns.
由于模型容量降低，超长序列性能下降。 Performance degrades for very long sequences due to reduced model capacity.
固定注意力模式需要针对不同数据类型仔细调整超参数。 Fixed attention patterns require careful hyperparameter tuning for different data types.
步长注意力在无周期结构的数据（如文本）上效果不佳。 Strided attention fails on data without periodic structure like text.
训练极深网络仍需仔细的初始化和架构修改。 Training very deep networks still requires careful initialization and architecture modifications.
论文章节 · Sections（共 23）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 背景3 Background
4 因子化自注意力4 Factorized Self-Attention
4.1 学习到的注意力模式的定性评估4.1 Qualitative assessment of learned attention patterns