We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness. To approximate softmax attention-kernels, Performers use a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+), which may be of independent interest for scalable kernel methods. FAVOR+ can be also used to efficiently model kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with other kernels for the first time on large-scale tasks, beyond the reach of regular Transformers, and investigate optimal attention-kernels. Performers are linear architectures fully compatible with regular Transformers and with strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix, uniform convergence and low estimation variance. We tested Performers on a rich set of tasks stretching from pixel-prediction through text models to protein sequence modeling. We demonstrate competitive results with other examined efficient sparse and dense attention methods, showcasing effectiveness of the novel attention-learning paradigm leveraged by Performers.
核心贡献 · Key contributions
提出 Performer，一种线性 Transformer 架构，能以可证明的精度估计 softmax 满秩注意力，且空间和时间复杂度为线性。 Introduces Performers, linear Transformer architectures that estimate softmax full-rank attention with provable accuracy using linear space and time complexity.
提出 FAVOR+机制，利用正正交随机特征实现 softmax 核的无偏、低方差近似。 Proposes FAVOR+ mechanism using positive orthogonal random features for unbiased, low-variance softmax kernel approximation.
提供理论保证：注意力矩阵估计的无偏性、一致收敛性和更低方差。 Provides theoretical guarantees: unbiased estimation, uniform convergence, and lower variance for attention matrix approximation.
在像素预测、文本建模和蛋白质序列建模等多样化任务上展示了竞争性能。 Demonstrates competitive performance on diverse tasks including pixel prediction, text modeling, and protein sequence modeling.
通过微调实现与预训练 Transformer 的向后兼容。 Enables backward compatibility with pre-trained Transformers via fine-tuning.
将注意力扩展到 softmax 之外的其他可核化机制，支持大规模核比较。 Extends attention beyond softmax to other kernelizable mechanisms, enabling large-scale kernel comparison.
局限 · Limitations
需要定期重绘随机特征以达到最佳训练效果，增加少量开销。 Requires periodic redrawing of random features for optimal training, adding minor overhead.
近似误差可能通过多层传播，深度模型需要更精确的近似。 Approximation errors may propagate through multiple layers, requiring tight approximation for deep models.
正交随机特征要求 m ≤ d，限制了特征数量不超过嵌入维度。 Orthogonal random features require m ≤ d, limiting the number of features to the embedding dimension.
在测试长度（如>12k 词元）之外的极长序列上的性能尚未充分验证。 Performance on very long sequences beyond tested lengths (e.g., >12k tokens) is not fully validated.
该方法假设各向同性分布用于正交化，可能不适用于所有核。 The method assumes isotropic distributions for orthogonalization, which may not hold for all kernels.
论文章节 · Sections（共 17）
摘要Abstract
1 引言与相关工作1 Introduction and related work
2 FAVOR+机制与正正交随机特征2 FAVOR+ Mechanism & Positive Orthogonal Random Features