Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. Notably, RoPE enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding. Finally, we evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets. Our experiments show that it consistently overcomes its alternatives. Furthermore, we provide a theoretical analysis to explain some experimental results. RoFormer is already integrated into Huggingface: \url{https://huggingface.co/docs/transformers/model_doc/roformer}.
核心贡献 · Key contributions
提出旋转位置编码（RoPE），通过旋转矩阵编码绝对位置，并通过内积编码相对位置。 Proposes Rotary Position Embedding (RoPE) that encodes absolute position via rotation matrix and relative position via inner product.
实现长期衰减特性，即词元间依赖随相对距离增加而衰减。 Achieves long-term decay property where inter-token dependency decreases with increasing relative distance.
兼容线性自注意力，无需二次复杂度即可实现相对位置编码。 Enables compatibility with linear self-attention, allowing relative position encoding without quadratic complexity.
在预训练中相比其他位置编码方法的基线模型收敛更快。 Demonstrates faster convergence in pre-training compared to baseline models with other position encoding methods.
在英文和中文长文本分类基准上优于替代方法。 Outperforms alternatives on long text classification benchmarks, including English and Chinese datasets.
提供理论分析解释 RoPE 的性质和有效性。 Provides theoretical analysis explaining the properties and effectiveness of RoPE.
局限 · Limitations
缺乏对相比其他位置编码策略更快收敛的透彻解释。 Lacks thorough explanation for faster convergence compared to other position encoding strategies.
尽管有长期衰减特性，但未对长文本优越性能给出可靠解释。 No faithful explanation for superior long-text performance despite long-term decay property.
基于 Transformer 架构，预训练需要大量硬件资源。 Requires significant hardware resources for pre-training due to Transformer-based infrastructure.
理论分析限于 2D 子空间，推广到更高维度依赖线性假设。 Theoretical analysis limited to 2D subspaces, generalization to higher dimensions relies on linearity assumption.
实验验证主要针对分类任务，其他 NLP 任务未广泛评估。 Experimental validation primarily on classification tasks; other NLP tasks not extensively evaluated.