Although using convolutional neural networks (CNNs) as backbones achieves great successes in computer vision, this work investigates a simple backbone network useful for many dense prediction tasks without convolutions. Unlike the recently-proposed Transformer model (e.g., ViT) that is specially designed for image classification, we propose Pyramid Vision Transformer~(PVT), which overcomes the difficulties of porting Transformer to various dense prediction tasks. PVT has several merits compared to prior arts. (1) Different from ViT that typically has low-resolution outputs and high computational and memory cost, PVT can be not only trained on dense partitions of the image to achieve high output resolution, which is important for dense predictions but also using a progressive shrinking pyramid to reduce computations of large feature maps. (2) PVT inherits the advantages from both CNN and Transformer, making it a unified backbone in various vision tasks without convolutions by simply replacing CNN backbones. (3) We validate PVT by conducting extensive experiments, showing that it boosts the performance of many downstream tasks, e.g., object detection, semantic, and instance segmentation. For example, with a comparable number of parameters, RetinaNet+PVT achieves 40.4 AP on the COCO dataset, surpassing RetinNet+ResNet50 (36.3 AP) by 4.1 absolute AP. We hope PVT could serve as an alternative and useful backbone for pixel-level predictions and facilitate future researches. Code is available at https://github.com/whai362/PVT.
核心贡献 · Key contributions
提出金字塔视觉 Transformer（PVT），首个用于密集预测任务的纯 Transformer 骨干网络。 Proposes Pyramid Vision Transformer (PVT), the first pure Transformer backbone for dense prediction tasks.
引入渐进式收缩金字塔以生成多尺度特征图，实现对密集预测的灵活适配。 Introduces progressive shrinking pyramid to generate multi-scale feature maps, enabling flexible adaptation to dense prediction.
设计空间缩减注意力（SRA）以降低高分辨率特征的计算和内存开销。 Designs spatial-reduction attention (SRA) to reduce computational and memory costs for high-resolution features.
在目标检测、实例分割和语义分割上以更少参数取得领先结果。 Achieves state-of-the-art results on object detection, instance segmentation, and semantic segmentation with fewer parameters.
证明 PVT 可在无卷积情况下替代 CNN 骨干用于多种视觉任务。 Demonstrates that PVT can replace CNN backbones in various vision tasks without convolutions.
通过结合 PVT 与 DETR 构建首个无卷积目标检测流程。 Builds first convolution-free object detection pipeline by combining PVT with DETR.
局限 · Limitations
PVT 在大输入尺度（如>640x640）下计算成本高于 ResNet。 PVT has higher computational cost than ResNet for large input scales (e.g., >640x640).
在 COCO（短边 800 像素）上，基于 PVT 的 RetinaNet 推理速度慢于基于 ResNet50 的模型。 Inference speed of PVT-based RetinaNet is slower than ResNet50-based model on COCO (800px short side).
未集成 SE、SK、空洞卷积和 NAS 等特定 CNN 模块。 Specific CNN modules like SE, SK, dilated convolution, and NAS are not integrated into PVT.
与 EfficientNet 等精心设计的 CNN 骨干相比，PVT 仍处于早期发展阶段。 PVT is still in early development stage compared to well-engineered CNN backbones like EfficientNet.
在 ImageNet 上预训练的位置嵌入在下游任务中需针对任意输入尺寸进行插值。 Position embeddings pre-trained on ImageNet may require interpolation for arbitrary input sizes in downstream tasks.
论文章节 · Sections（共 20）
摘要Abstract
1 引言1 Introduction
2.1 CNN 骨干网络2.1 CNN Backbones
2.2 密集预测任务2.2 Dense Prediction Tasks
2.3 视觉中的自注意力与 Transformer2.3 Self-Attention and Transformer in Vision
3.1 整体架构3.1 Overall Architecture
3.2 用于 Transformer 的特征金字塔3.2 Feature Pyramid for Transformer
3.3 Transformer 编码器3.3 Transformer Encoder
3.4 模型细节3.4 Model Details
3.5 讨论3.5 Discussion
4.1 图像级预测4.1 Image-Level Prediction
4.2 像素级密集预测4.2 Pixel-Level Dense Prediction
5 实验5 Experiments
5.1 图像分类5.1 Image Classification
5.2 目标检测5.2 Object Detection
5.3 语义分割5.3 Semantic Segmentation
5.4 纯 Transformer 检测与分割5.4 Pure Transformer Detection & Segmentation