一张图片等价于 16x16 个词：用于大规模图像识别的 Transformer

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

阿列克谢·多索维茨基 Alexey Dosovitskiy · Google · 2020-10-22 · arXiv:2010.11929 ↗ · 被引 65359

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

虽然 Transformer 架构已成为自然语言处理任务的事实标准，但其在计算机视觉中的应用仍然有限。在视觉领域，注意力机制要么与卷积网络结合使用，要么用于替换卷积网络的某些组件，同时保留其整体结构。我们表明，这种对 CNN 的依赖并非必要，直接将纯 Transformer 应用于图像块序列可以在图像分类任务上表现非常出色。当在大量数据上预训练并迁移到多个中、小规模图像识别基准（ImageNet、CIFAR-100、VTAB 等）时，Vision Transformer（ViT）相比最先进的卷积网络取得了优异的结果，同时训练所需计算资源大幅减少。

While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

核心贡献 · Key contributions

• 展示了将纯 Transformer 直接应用于图像块可实现强大的图像分类性能。
  Shows pure Transformer applied to image patches achieves strong image classification performance.
• 证明了 Vision Transformer（ViT）在大规模预训练上匹配或超越最先进的 CNN。
  Demonstrates Vision Transformer (ViT) matches or exceeds state-of-the-art CNNs on large-scale pre-training.
• 揭示了大规模训练克服了 ViT 中归纳偏置的缺乏。
  Reveals large-scale training overcomes lack of inductive biases in ViT.
• 提供了受控的规模扩张研究，显示 ViT 有利的性能成本比。
  Provides controlled scaling study showing ViT's favorable performance-to-cost ratio.
• 分析了 ViT 内部表示，展示了全局注意力和学习到的位置嵌入。
  Analyzes ViT internal representations, showing global attention and learned position embeddings.
• 探索了 ViT 的自监督预训练，展示了未来改进的潜力。
  Explores self-supervised pre-training for ViT, showing potential for future improvement.

局限 · Limitations

• 由于缺乏归纳偏置，ViT 在小型数据集上表现不如 CNN。
  ViT underperforms CNNs on small datasets due to lack of inductive biases.
• 需要大规模预训练（如 JFT-300M）才能达到最先进结果。
  Requires large-scale pre-training (e.g., JFT-300M) to achieve state-of-the-art results.
• 自监督预训练在 ImageNet 上仍落后监督预训练 4%。
  Self-supervised pre-training still lags behind supervised pre-training by 4% on ImageNet.
• ViT 尚未应用于检测和分割等其他视觉任务。
  ViT has not been applied to other vision tasks like detection and segmentation.
• 位置嵌入需要插值以进行更高分辨率的微调。
  Position embeddings require interpolation for higher resolution fine-tuning.

论文章节 · Sections（共 15）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2 相关工作 2 Related Work
• 3 方法 3 Method
• 3.1 视觉变换器（ViT） 3.1 Vision Transformer (ViT)
• 3.2 微调与更高分辨率 3.2 Fine-tuning and Higher Resolution
• 4 实验 4 Experiments
• 4.1 设置 4.1 Setup
• 4.2 与最新技术的比较 4.2 Comparison to State of the Art
• 4.3 预训练数据需求 4.3 Pre-training Data Requirements
• 4.4 规模研究 4.4 Scaling Study
• 4.5 视觉变换器分析 4.5 Inspecting Vision Transformer
• 4.6 自监督 4.6 Self-supervision
• 5 结论 5 Conclusion
• 致谢 Acknowledgements

阅读逐段中英对照全文 →