PVT v2：改进的金字塔视觉 Transformer 基线

PVT v2: Improved Baselines with Pyramid Vision Transformer

王文海 Wenhai Wang · Nanjing University / Shanghai AI Lab · 2021-06-25 · arXiv:2106.13797 ↗ · 被引 2465

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

Transformer 最近在计算机视觉领域取得了令人鼓舞的进展。在这项工作中，我们通过改进原始的金字塔视觉 Transformer（PVT v1），添加了三种设计，包括（1）线性复杂度注意力层，（2）重叠块嵌入，以及（3）卷积前馈网络，提出了新的基线。通过这些修改，PVT v2 将 PVT v1 的计算复杂度降低到线性，并在分类、检测和分割等基本视觉任务上取得了显著改进。值得注意的是，所提出的 PVT v2 取得了与 Swin Transformer 等近期工作相当或更好的性能。我们希望这项工作能促进计算机视觉领域最先进的 Transformer 研究。代码可在 https://github.com/whai362/PVT 获取。

Transformer recently has presented encouraging progress in computer vision. In this work, we present new baselines by improving the original Pyramid Vision Transformer (PVT v1) by adding three designs, including (1) linear complexity attention layer, (2) overlapping patch embedding, and (3) convolutional feed-forward network. With these modifications, PVT v2 reduces the computational complexity of PVT v1 to linear and achieves significant improvements on fundamental vision tasks such as classification, detection, and segmentation. Notably, the proposed PVT v2 achieves comparable or better performances than recent works such as Swin Transformer. We hope this work will facilitate state-of-the-art Transformer researches in computer vision. Code is available at https://github.com/whai362/PVT.

核心贡献 · Key contributions

• 提出使用平均池化的线性空间缩减注意力层，将复杂度降至线性。
  Proposes linear spatial reduction attention (SRA) layer with average pooling, reducing complexity to linear.
• 引入重叠补丁嵌入以建模图像的局部连续性。
  Introduces overlapping patch embedding to model local continuity of images.
• 添加带零填充的卷积前馈网络以捕获局部连续性并移除固定位置编码。
  Adds convolutional feed-forward network with zero-padding to capture local continuity and remove fixed position encoding.
• 在 ImageNet 分类、COCO 检测和 ADE20K 分割上取得最先进结果。
  Achieves state-of-the-art results on ImageNet classification, COCO detection, and ADE20K segmentation.
• 提供一系列不同规模的模型（B0-B5），性能优于 Swin Transformer 和 Twins。
  Provides a family of models (B0-B5) with varying sizes, outperforming Swin Transformer and Twins.

局限 · Limitations

• 评估仅限于图像分类、目标检测和语义分割任务。
  Evaluation limited to image classification, object detection, and semantic segmentation tasks.
• 在超出测试尺度的极高分辨率输入任务上性能可能下降。
  Performance may degrade on tasks requiring very high-resolution inputs beyond tested scales.
• 消融研究仅在 PVT v2-B2 变体上进行；其他规模可能表现出不同的权衡。
  Ablation studies only on PVT v2-B2 variant; other sizes may show different trade-offs.
• 未探索与其他先进架构（如基于 MLP 的主干网络）的集成。
  Does not explore integration with other advanced architectures like MLP-based backbones.

论文章节 · Sections（共 14）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2 相关工作 2 Related Work
• 3.1 PVT v1 的局限性 3.1 Limitations in PVT v1
• 3.2 线性空间缩减注意力 3.2 Linear Spatial Reduction Attention
• 3.3 重叠块嵌入 3.3 Overlapping Patch Embedding
• 3.4 卷积前馈网络 3.4 Convolutional Feed-Forward
• 3.5 PVT v2 系列细节 3.5 Details of PVT v2 Series
• 3.6 PVT v2 的优势 3.6 Advantages of PVT v2
• 4.1 图像分类 4.1 Image Classification
• 4.2 目标检测 4.2 Object Detection
• 4.3 语义分割 4.3 Semantic Segmentation
• 4.4 消融实验 4.4 Ablation Study
• 5 结论 5 Conclusion

阅读逐段中英对照全文 →