Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization dataset.
核心贡献 · Key contributions
提出 Longformer，其注意力机制线性缩放，适用于长文档处理。 Introduces Longformer with linear-scaling attention for long documents.
结合局部窗口注意力和任务驱动的全局注意力。 Combines local windowed attention with task-specific global attention.
在 text8 和 enwik8 字符级语言建模上达到最优结果。 Achieves state-of-the-art on text8 and enwik8 character-level language modeling.
预训练 Longformer 并在下游任务微调，长文档任务上超越 RoBERTa。 Pretrains Longformer and fine-tunes on downstream tasks, outperforming RoBERTa on long documents.
在 WikiHop 和 TriviaQA 上取得新最优结果。 Sets new state-of-the-art on WikiHop and TriviaQA.
提出 Longformer 编码器-解码器（LED），用于长文档序列到序列任务。 Introduces Longformer-Encoder-Decoder (LED) for long-document seq2seq tasks.
局限 · Limitations
注意力模式需要针对任务配置全局注意力。 Attention pattern requires task-specific global attention configuration.
从 RoBERTa 检查点预训练可能限制对新模式的适应。 Pretraining from RoBERTa checkpoint may limit adaptation to new patterns.
膨胀注意力与预训练 RoBERTa 权重不兼容。 Dilated attention not compatible with pretrained RoBERTa weights.
短文档上性能与基线相似，提升有限。 Performance on short documents similar to baselines, limited gain.
膨胀注意力的自定义 CUDA 内核实现复杂。 Custom CUDA kernel implementation complexity for dilated attention.
论文章节 · Sections（共 18）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 Longformer3 Longformer
3.1 注意力模式3.1 Attention Pattern
3.2 实现3.2 Implementation
4 自回归语言建模4 Autoregressive Language Modeling
4.1 注意力模式4.1 Attention Pattern
4.2 实验设置4.2 Experiment Setup
5 预训练与微调5 Pretraining and Finetuning
6 任务6 Tasks
6.1 问答6.1 Question answering
6.2 指代消解6.2 Coreference Resolution
6.3 文档分类6.3 Document Classification
6.4 结果6.4 Results
附录 B 字符级语言模型超参数Appendix B Character LM Hyperparameters
附录 C 预训练数据Appendix C Pretraining Data
附录 D 任务特定模型细节Appendix D Task specific model details