本文提出了一种新的视觉 Transformer——Swin Transformer，它能够作为计算机视觉的通用骨干网络。将 Transformer 从语言领域适应到视觉领域面临挑战，原因在于两个领域的差异，例如视觉实体尺度的巨大变化以及图像中像素的高分辨率（相较于文本中的词）。为了解决这些差异，我们提出了一种分层 Transformer，其表示通过移位窗口计算。移位窗口方案通过将自注意力计算限制在非重叠的局部窗口内，同时允许跨窗口连接，从而提高了效率。这种分层架构具有在不同尺度上建模的灵活性，并且计算复杂度与图像大小呈线性关系。Swin Transformer 的这些特性使其适用于广泛的视觉任务，包括图像分类（ImageNet-1K 上 top-1 准确率 87.3%）和密集预测任务，如目标检测（COCO test-dev 上 box AP 58.7 和 mask AP 51.1）以及语义分割（ADE20K val 上 mIoU 53.5）。其性能在 COCO 上以+2.7 box AP 和+2.6 mask AP 的大幅优势超越了之前的最优结果，在 ADE20K 上以+3.2 mIoU 超越，展示了基于 Transformer 的模型作为视觉骨干网络的潜力。分层设计和移位窗口方法也被证明对所有 MLP 架构有益。代码和模型已在 https://github.com/microsoft/Swin-Transformer 公开。
This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with \textbf{S}hifted \textbf{win}dows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at~\url{https://github.com/microsoft/Swin-Transformer}.
核心贡献 · Key contributions
提出 Swin Transformer，一种采用移位窗口的分层视觉 Transformer，作为通用骨干网络。 Proposes Swin Transformer, a hierarchical vision Transformer with shifted windows for general-purpose backbone.
通过局部窗口自注意力实现与图像大小成线性关系的计算复杂度。 Achieves linear computational complexity relative to image size via local window self-attention.
引入移位窗口划分，高效实现跨窗口连接。 Introduces shifted window partitioning to enable cross-window connections efficiently.