Since the introduction of the transformer model by Vaswani et al. (2017), a fundamental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training? We first show that extrapolation can be enabled by simply changing the position representation method, though we find that current methods do not allow for efficient extrapolation. We therefore introduce a simpler and more efficient position method, Attention with Linear Biases (ALiBi). ALiBi does not add positional embeddings to word embeddings; instead, it biases query-key attention scores with a penalty that is proportional to their distance. We show that this method trains a 1.3 billion parameter model on input sequences of length 1024 that extrapolates to input sequences of length 2048, achieving the same perplexity as a sinusoidal position embedding model trained on inputs of length 2048 but training 11% faster and using 11% less memory. ALiBi's inductive bias towards recency also leads it to outperform multiple strong position methods on the WikiText-103 benchmark.
核心贡献 · Key contributions
提出 ALiBi，一种简单的注意力位置方法，通过线性惩罚偏置注意力分数，惩罚与距离成正比。 Introduces ALiBi, a simple position method that biases attention scores with linear penalties proportional to distance.
展示 ALiBi 使 Transformer 语言模型能外推到比训练长度更长的序列，例如 1.3B 参数模型在 1024 词元上训练可外推到 2048 词元。 Shows ALiBi enables transformer LMs to extrapolate to sequences longer than training length, e.g., 1.3B model trained on 1024 tokens extrapolates to 2048.
证明 ALiBi 在长序列上达到与正弦位置嵌入相同的困惑度时，训练速度快 11%，内存少 11%。 Demonstrates ALiBi trains 11% faster and uses 11% less memory than sinusoidal embeddings for equivalent perplexity on long sequences.
发现外推能力高度依赖位置方法；正弦和旋转方法外推有限，而 T5 偏置计算成本高。 Finds that extrapolation ability depends heavily on position method; sinusoidal and rotary methods have limited extrapolation, while T5 bias is computationally costly.
展示 ALiBi 在 WikiText-103 上优于多种强位置方法，困惑度在训练长度 3 倍内持续改善。 Shows ALiBi outperforms multiple strong position methods on WikiText-103, improving perplexity up to 3x training length.
在 461GB 数据集上验证 1.3B 参数模型的 ALiBi，以更少内存达到与正弦基线相似的困惑度。 Validates ALiBi on large-scale 1.3B parameter model with 461GB dataset, achieving similar perplexity to sinusoidal baseline with less memory.
局限 · Limitations
ALiBi 的外推性能在训练长度约 2 倍时达到峰值，之后增益递减。 ALiBi's extrapolation performance peaks at around 2x training length; beyond that, gains diminish.
方法主要在语言建模上测试；未探索在翻译或分类等其他任务上的适用性。 Method tested primarily on language modeling; applicability to other tasks like translation or classification not explored.
ALiBi 因每头偏置掩码需要少量内存增加（最多 100MB），但被更短的训练序列抵消。 ALiBi requires a small memory increase (up to 100MB) due to per-head bias masks, though offset by shorter training sequences.
固定斜率超参数集可能并非对所有模型规模或领域最优，但在测试设置中表现稳健。 The fixed slope hyperparameter set may not be optimal for all model sizes or domains, though robust across tested settings.
外推到超过 10,000 词元时性能强但不完美；进一步改进可能需要额外技术。 Extrapolation beyond 10,000 tokens shows strong but not perfect performance; further improvements may require additional techniques.
论文章节 · Sections（共 11）
摘要Abstract
1 引言1 Introduction
2 现有方法无法高效外推2 Current Approaches Do Not Extrapolate Efficiently
2.1 背景与实验设置2.1 Background and Experimental Setup
2.2 外推测量2.2 Measuring Extrapolation
3 带线性偏置的注意力机制（ALiBi）3 Attention with Linear Biases (ALiBi)
4 结果4 Results
4.1 WikiText-103 和 Toronto BookCorpus 上的结果4.1 Results on WikiText-103 and Toronto BookCorpus
4.2 CC100+RoBERTa 语料库上的结果4.2 Results on the CC100+RoBERTa Corpus