We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries, DETR reasons about the relations of the objects and the global image context to directly output the final set of predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive baselines. Training code and pretrained models are available at https://github.com/facebookresearch/detr.
核心贡献 · Key contributions
提出 DETR，一种基于 Transformer 的端到端目标检测器，将检测视为直接集合预测问题。 Proposes DETR, a transformer-based end-to-end object detector that treats detection as a direct set prediction problem.
通过二分匹配损失消除了非极大值抑制和锚点生成等手工设计组件。 Eliminates hand-designed components like non-maximum suppression and anchor generation via bipartite matching loss.
在 COCO 上实现了与 Faster R-CNN 相当的精度和运行时间，在大目标上表现更优。 Achieves competitive accuracy and runtime on COCO compared to Faster R-CNN, with superior performance on large objects.
通过统一掩码头自然扩展到全景分割，优于强基线。 Extends naturally to panoptic segmentation with a unified mask head, outperforming strong baselines.
提供简单灵活的架构，可使用标准 CNN 和 Transformer 库实现。 Provides a simple and flexible architecture that can be implemented with standard CNN and transformer libraries.
局限 · Limitations
需要超长训练周期（300-500 轮）和辅助损失才能收敛。 Requires extra-long training schedule (300-500 epochs) and auxiliary losses for convergence.
在小目标上性能不如 Faster R-CNN，尤其在没有空洞骨干时。 Performs worse on small objects compared to Faster R-CNN, especially without dilated backbone.
由于 Transformer 自注意力计算成本，推理速度慢于某些优化检测器。 Inference speed is slower than some optimized detectors due to transformer self-attention cost.
固定数量的目标查询限制了每张图像的最大检测数，可能遗漏大量目标。 Fixed number of object queries limits maximum detections per image, potentially missing many objects.
论文章节 · Sections（共 16）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related work
2.1 集合预测2.1 Set Prediction
2.2 Transformer 与并行解码2.2 Transformers and Parallel Decoding
2.3 目标检测2.3 Object detection
3 DETR 模型3 The DETR model
3.1 目标检测集合预测损失3.1 Object detection set prediction loss
3.2 DETR 架构3.2 DETR architecture
4 实验4 Experiments
4.1 与 Faster R-CNN 对比4.1 Comparison with Faster R-CNN
4.2 消融实验4.2 Ablations
4.3 分析4.3 Analysis
4.4 DETR 在全景分割中的应用4.4 DETR for panoptic segmentation