ImageBind：一个嵌入空间绑定所有

ImageBind: One Embedding Space To Bind Them All

罗斯·吉尔希克 Ross Girshick · · 2023-05-09 · arXiv:2305.05665 ↗ · 被引 1660

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

我们提出 ImageBind，一种学习跨六种不同模态（图像、文本、音频、深度、热成像和 IMU 数据）的联合嵌入的方法。我们表明，训练这样的联合嵌入并不需要所有配对数据的组合，仅图像配对数据就足以将模态绑定在一起。ImageBind 可以利用最近的大规模视觉-语言模型，并通过与图像的自然配对将其零样本能力扩展到新模态。它实现了开箱即用的新涌现应用，包括跨模态检索、用算术组合模态、跨模态检测和生成。涌现能力随着图像编码器的强度而提高，我们在跨模态的涌现零样本识别任务上达到了新的最先进水平，优于专业监督模型。最后，我们展示了强大的少样本识别结果，优于先前的工作，并且 ImageBind 作为评估视觉模型在视觉和非视觉任务上的新方法。

We present ImageBind, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. ImageBind can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications 'out-of-the-box' including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models. Finally, we show strong few-shot recognition results outperforming prior work, and that ImageBind serves as a new way to evaluate vision models for visual and non-visual tasks.

核心贡献 · Key contributions

• 仅通过图像配对数据即可绑定六种模态，无需所有模态对数据。
  Binds six modalities via image-paired data alone, without requiring all modality pairs.
• 利用大规模视觉-语言模型将零样本能力扩展到新模态。
  Leverages large-scale vision-language models to extend zero-shot capabilities to new modalities.
• 实现开箱即用的新兴跨模态检索、算术、检测和生成。
  Enables emergent cross-modal retrieval, arithmetic, detection, and generation out-of-the-box.
• 在新兴零样本跨模态识别任务上设立新 SOTA，超越监督模型。
  Sets new state-of-the-art on emergent zero-shot recognition across modalities, outperforming supervised models.
• 强少样本识别结果，可作为评估视觉模型的新工具。
  Strong few-shot recognition results and serves as a new evaluation tool for vision models.

局限 · Limitations

• 依赖图像配对数据；弱或错位图像对会降低绑定质量。
  Relies on image-paired data; binding quality degrades with weak or misaligned image pairs.
• 需要大规模预训练视觉-语言模型，导致高计算成本。
  Requires large pretrained vision-language models, leading to high computational cost.
• 非视觉模态性能仍落后于专用监督模型。
  Non-visual modality performance still lags behind specialist supervised models.
• 深度、热成像和 IMU 数据集规模小，影响小编码器性能。
  Limited by small-scale datasets for depth, thermal, and IMU, affecting small encoders.
• 强烈依赖空间/时间对齐；错对齐导致性能显著下降。
  Strongly dependent on spatial/temporal alignment; misalignment causes significant performance drop.

论文章节 · Sections（共 16）

• 摘要 Abstract
• 1 引言 1 Introduction
• 相关工作 2 Related Work
• 3 方法 3 Method
• 3.1 预备知识 3.1 Preliminaries
• 3.2 将模态与图像绑定 3.2 Binding modalities with images
• 3.3 实现细节 3.3 Implementation Details
• 4 实验 4 Experiments
• 4.1 涌现式零样本分类 4.1 Emergent zero-shot classification
• 4.2 与先前工作比较 4.2 Comparison to prior work
• 4.3 少样本分类 4.3 Few-shot classification
• 4.4 分析与应用 4.4 Analysis and Applications
• 5 消融实验 5 Ablation Study
• 5.1 缩放图像编码器 5.1 Scaling the Image Encoder
• 5.2 训练损失与架构 5.2 Training Loss and Architecture
• 6 讨论与局限性 6 Discussion and Limitations

阅读逐段中英对照全文 →