Transformer 家族版本 2.0

The Transformer Family Version 2.0

翁荔 Lilian Weng · · 2023-01-27 · Article ↗

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

日期：2023 年 1 月 27 日 | 预计阅读时间：45 分钟 | 作者：Lilian Weng * 局部与全局上下文的结合 * 用于强化学习的 Transformer

Date: January 27, 2023 | Estimated Reading Time: 45 min | Author: Lilian Weng * Combination of Local and Global Context * Transformers for Reinforcement Learning

核心贡献 · Key contributions

• 对 2020 年以来 Transformer 架构变体与改进进行了全面、结构化的综述。
  Provides a comprehensive structured survey of Transformer architecture variants and improvements since 2020.
• 系统梳理了注意力机制与位置编码，包括相对位置编码和旋转位置嵌入。
  Systematically reviews attention mechanisms and positional encodings, including relative position encoding and rotary position embedding.
• 讨论了通过记忆与检索扩展上下文窗口的机制，如 Transformer-XL、Compressive Transformer、kNN-LM 和 Memorizing Transformer。
  Discusses mechanisms for extending context window via memory and retrieval, such as Transformer-XL, Compressive Transformer, kNN-LM, and Memorizing Transformer.
• 涵盖了自适应计算方法，包括 Universal Transformer、自适应注意力跨度、深度自适应 Transformer 和 CALM。
  Covers adaptive computation methods including Universal Transformer, adaptive attention span, depth-adaptive Transformer, and CALM.
• 综述了稀疏、局部敏感哈希、低秩和随机特征注意力等效率改进，以降低二次复杂度。
  Surveys efficiency improvements like sparse, LSH, low-rank, and random feature attention to reduce quadratic complexity.
• 重点介绍了局部-全局混合注意力模型，包括 ETC、Longformer 和 Big Bird。
  Highlights hybrid local-global attention models including ETC, Longformer, and Big Bird.

局限 · Limitations

• 该综述是一个快照；Transformer 研究发展迅速，其覆盖范围和框架可能很快过时。
  The survey is a snapshot; Transformer research evolves rapidly, so coverage and frameworks may become outdated.
• 许多技术仅在特定任务和数据集上评估，其相对收益可能无法跨领域或跨模型规模迁移。
  Many techniques are evaluated on specific tasks and datasets, so their relative gains may not transfer across domains or model scales.
• 高效注意力方法常常以精度换取速度和内存，引入近似误差或限制上下文范围。
  Efficient attention methods often trade off accuracy for speed and memory, introducing approximation errors or restricted context.
• ALiBi 等长上下文外推方法的泛化能力有限，在更长序列上可能性能下降。
  Long-context extrapolation methods like ALiBi show limited extrapolation and may degrade on much longer sequences.
• 基于记忆和检索的增强方法需要额外的存储和计算，在极大规模下可能不切实际。
  Memory and retrieval based augmentations require extra storage and computation, which may be impractical at very large scale.

论文章节 · Sections（共 25）

• 概述 Overview
• Transformer 基础 Transformer Basics#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#transformer-basics)
• 注意力与自注意力 Attention and Self-Attention#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#attention-and-self-attention)
• 多头自注意力 Multi-Head Self-Attention#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#multi-head-self-attention)
• 编码器-解码器架构 Encoder-Decoder Architecture#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#encoder-decoder-architecture)
• 位置编码 Positional Encoding#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#positional-encoding)
• 正弦位置编码 Sinusoidal Positional Encoding#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#sinusoidal-positional-encoding)
• 可学习位置编码 Learned Positional Encoding#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#learned-positional-encoding)
• 相对位置编码 Relative Position Encoding#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#relative-position-encoding)
• 旋转位置嵌入 Rotary Position Embedding#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#rotary-position-embedding)
• 更长上下文 Longer Context#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#longer-context)
• 上下文记忆 Context Memory#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#context-memory)
• 不可微外部记忆 Non-Differentiable External Memory#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#non-differentiable-external-memory)
• 距离增强注意力分数 Distance-Enhanced Attention Scores#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#distance-enhanced-attention-scores)
• 循环化 Make it Recurrent#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#make-it-recurrent)
• 自适应建模 Adaptive Modeling#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#adaptive-modeling)
• 自适应注意力跨度 Adaptive Attention Span#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#adaptive-attention-span)
• 深度自适应 Transformer Depth-Adaptive Transformer#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#depth-adaptive-transformer)
• 高效注意力 Efficient Attention#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#efficient-attention)
• 固定局部上下文 Fixed Local Context#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#fixed-local-context)
• 跨步上下文 Strided Context#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#strided-context)
• 局部与全局上下文的组合 Combination of Local and Global Context#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#combination-of-local-and-global-context)
• 基于内容的注意力 Content-based Attention#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#content-based-attention)
• 低秩注意力 Low-Rank Attention#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#low-rank-attention)
• 强化学习中的 Transformer Transformers for Reinforcement Learning#(https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/#transformers-for-reinforcement-learning)

阅读逐段中英对照全文 →