High-resolution dense prediction enables many appealing real-world applications, such as computational photography, autonomous driving, etc. However, the vast computational cost makes deploying state-of-the-art high-resolution dense prediction models on hardware devices difficult. This work presents EfficientViT, a new family of high-resolution vision models with novel multi-scale linear attention. Unlike prior high-resolution dense prediction models that rely on heavy softmax attention, hardware-inefficient large-kernel convolution, or complicated topology structure to obtain good performances, our multi-scale linear attention achieves the global receptive field and multi-scale learning (two desirable features for high-resolution dense prediction) with only lightweight and hardware-efficient operations. As such, EfficientViT delivers remarkable performance gains over previous state-of-the-art models with significant speedup on diverse hardware platforms, including mobile CPU, edge GPU, and cloud GPU. Without performance loss on Cityscapes, our EfficientViT provides up to 13.9$\times$ and 6.2$\times$ GPU latency reduction over SegFormer and SegNeXt, respectively. For super-resolution, EfficientViT delivers up to 6.4x speedup over Restormer while providing 0.11dB gain in PSNR. For Segment Anything, EfficientViT delivers 48.9x higher throughput on A100 GPU while achieving slightly better zero-shot instance segmentation performance on COCO.
核心贡献 · Key contributions
提出多尺度线性注意力机制，以硬件高效操作实现全局感受野和多尺度学习。 Proposes multi-scale linear attention achieving global receptive field and multi-scale learning with hardware-efficient operations.
首次证明线性注意力在高分辨率密集预测中的有效性。 First to demonstrate effectiveness of linear attention for high-resolution dense prediction.
设计 EfficientViT 模型系列，在多种硬件平台上实现显著加速。 Designs EfficientViT model family with significant speedup on diverse hardware platforms.
在 Cityscapes 上相比 SegFormer 实现高达 13.9 倍 GPU 延迟降低，且无性能损失。 Achieves up to 13.9x GPU latency reduction on Cityscapes over SegFormer without performance loss.
在 Segment Anything 任务中，A100 GPU 吞吐量提升 48.9 倍，零样本实例分割性能更优。 Delivers 48.9x higher throughput on A100 GPU for Segment Anything with better zero-shot instance segmentation.
超分辨率任务相比 Restormer 实现高达 6.4 倍加速，PSNR 提升 0.11dB。 Provides up to 6.4x speedup over Restormer for super-resolution with 0.11dB PSNR gain.
局限 · Limitations
与 softmax 注意力相比，多尺度线性注意力在局部信息提取方面能力有限。 Multi-scale linear attention may have limited capacity for local information compared to softmax attention.
在 LVIS 单点提示分割上性能不如 SAM-ViT-H，需进一步研究。 Performance on single-point prompt segmentation in LVIS is inferior to SAM-ViT-H, requiring further investigation.
模型仅在特定任务（语义分割、超分辨率、Segment Anything）和数据集上评估。 Models are evaluated only on specific tasks (semantic segmentation, super-resolution, Segment Anything) and datasets.
Segment Anything 训练需两阶段流程，增加了复杂性。 Training requires two-phase process for Segment Anything, adding complexity.
未探索在视频或 3D 处理等其他视觉任务上的有效性。 Effectiveness on other vision tasks like video or 3D processing is not explored.