Grounding DINO：将 DINO 与接地预训练相结合，实现开放集目标检测

Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

张磊 Lei Zhang · IDEA Research · 2023-03-09 · arXiv:2303.05499 ↗ · 被引 4414

摘要 · Abstract

本文提出了一种名为 Grounding DINO 的开放集目标检测器，通过将基于 Transformer 的检测器 DINO 与接地预训练相结合，能够根据类别名称或指代表达等人类输入检测任意物体。开放集目标检测的关键解决方案是将语言引入封闭集检测器以实现开放集概念泛化。为了有效融合语言和视觉模态，我们将封闭集检测器概念性地分为三个阶段，并提出了一种紧密融合方案，包括特征增强器、语言引导的查询选择和用于跨模态融合的跨模态解码器。以往的工作主要在新颖类别上评估开放集目标检测，我们提出还对具有属性指定的物体的指代表达理解进行评估。Grounding DINO 在所有三种设置上均表现出色，包括 COCO、LVIS、ODinW 和 RefCOCO/+/g 基准测试。Grounding DINO 在 COCO 检测零样本迁移基准上达到了 52.5 AP，即未使用任何 COCO 训练数据。它在 ODinW 零样本基准上创下了新纪录，平均 AP 为 26.1。代码将在 https://github.com/IDEA-Research/GroundingDINO 提供。

In this paper, we present an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training, which can detect arbitrary objects with human inputs such as category names or referring expressions. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion. While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a $52.5$ AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean $26.1$ AP. Code will be available at \url{https://github.com/IDEA-Research/GroundingDINO}.

核心贡献 · Key contributions

提出 Grounding DINO，通过紧密的视觉-语言融合扩展 DINO，实现开放集检测。
Proposes Grounding DINO, extending DINO with tight vision-language fusion for open-set detection.
引入特征增强器、语言引导的查询选择和跨模态解码器，实现深度融合。
Introduces a feature enhancer, language-guided query selection, and cross-modality decoder for deep fusion.
将开放集检测评估扩展到指代表达理解（REC）数据集。
Extends open-set detection evaluation to referring expression comprehension (REC) datasets.
在 COCO 零样本迁移上达到 52.5 AP，在 ODinW 零样本基准上达到 26.1 平均 AP。
Achieves 52.5 AP on COCO zero-shot transfer and 26.1 mean AP on ODinW zero-shot benchmark.
提出子句级文本提示，避免不必要的词交互。
Proposes sub-sentence level text prompts to avoid unwanted word interactions.

局限 · Limitations

无法执行像 GLIPv2 那样的分割任务。
Cannot perform segmentation tasks like GLIPv2.
训练数据少于最大的 GLIP 模型，可能限制性能。
Training data is less than the largest GLIP model, potentially limiting performance.
900 个查询的设计可能限制了对 LVIS 中长尾物体的能力。
900 query design may limit ability for long-tailed objects in LVIS.
在没有 REC 数据微调的情况下，在 REC 任务上表现不佳。
Performance on REC tasks is poor without fine-tuning on REC data.

论文章节 · Sections（共 16）

摘要 Abstract
1 引言 1 Introduction
2 相关工作 2 Related Work
3 Grounding DINO 3 Grounding DINO
3.1 特征提取与增强器 3.1 Feature Extraction and Enhancer
3.2 语言引导的查询选择 3.2 Language-Guided Query Selection
3.3 跨模态解码器 3.3 Cross-Modality Decoder
3.4 子句级文本特征 3.4 Sub-Sentence Level Text Feature
3.5 损失函数 3.5 Loss Function
4.1 实验设置 4.1 Setup
4.2 Grounding DINO 的零样本迁移 4.2 Zero-Shot Transfer of Grounding DINO
4.3 指代目标检测设置 4.3 Referring Object Detection Settings
4.4 消融实验 4.4 Ablations
4.5 从 DINO 到 Grounding DINO 的迁移 4.5 Transfer from DINO to Grounding DINO
5 结论 5 Conclusion
6 致谢 6 Acknowledgement

阅读逐段中英对照全文 →