In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost $O(1)$ inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity, where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference. The intriguing properties make RetNet a strong successor to Transformer for large language models. Code will be available at https://aka.ms/retnet.
核心贡献 · Key contributions
提出 Retentive Network（RetNet）作为大型语言模型的基础架构，同时实现训练并行性、低成本推理和良好性能。 Proposes Retentive Network (RetNet) as a foundation architecture for large language models, achieving training parallelism, low-cost inference, and good performance.
引入多尺度保留机制，支持并行、循环和分块循环三种计算范式。 Introduces a multi-scale retention mechanism with three computation paradigms: parallel, recurrent, and chunkwise recurrent.
从理论上推导了循环与注意力机制之间的联系，实现了双形式序列建模。 Theoretically derives the connection between recurrence and attention, enabling dual-form sequence modeling.
展示了 O(1)推理复杂度，与 Transformer 相比显著降低内存和延迟。 Demonstrates O(1) inference complexity, significantly reducing memory and latency compared to Transformers.
在语言建模中取得有竞争力的性能，并在不同模型规模下呈现良好的缩放结果。 Achieves competitive language modeling performance and favorable scaling results across model sizes.
通过线性复杂度的分块循环表示实现高效的长序列建模。 Provides efficient long-sequence modeling via chunkwise recurrent representation with linear complexity.
局限 · Limitations
RetNet 的训练效率依赖于分块循环表示，可能需要仔细调整块大小。 RetNet's training efficiency relies on chunkwise recurrent representation, which may require careful tuning of chunk size.
与 Transformer 相比，性能提升在较大模型规模时更显著；较小模型可能获益较少。 Performance gains over Transformers are more pronounced at larger model sizes; smaller models may not benefit as much.
研究未探索超过 6.7B 参数或 100B token 的训练规模。 The study does not explore scaling beyond 6.7B parameters or training beyond 100B tokens.
RetNet 的推理速度优势在特定硬件上展示；推广到其他平台可能有所不同。 RetNet's inference speed advantage is demonstrated on specific hardware; generalization to other platforms may vary.
论文章节 · Sections（共 16）
摘要Abstract
1 引言1 Introduction
2 保持网络2 Retentive Networks
2.1 保持机制2.1 Retention
2.2 门控多尺度保持2.2 Gated Multi-Scale Retention
2.3 保持网络的整体架构2.3 Overall Architecture of Retention Networks
2.4 与先前方法的关系与区别2.4 Relation to and Differences from Previous Methods
3 实验3 Experiments
3.1 实验设置3.1 Setup
3.2 与 Transformer 的比较3.2 Comparisons with Transformer
3.3 训练成本3.3 Training Cost
3.4 推理成本3.4 Inference Cost
3.5 与 Transformer 变体的比较3.5 Comparison with Transformer Variants