利用噪声文本监督扩展视觉和视觉-语言表示学习

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

贾超 Chao Jia · Google · 2021-02-11 · arXiv:2102.05918 ↗ · 被引 5678

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

预训练表示对许多自然语言处理和感知任务至关重要。虽然自然语言处理中的表示学习已转向无需人工标注的原始文本训练，但视觉和视觉-语言表示仍严重依赖昂贵或需要专业知识的精心策划数据集。对于视觉应用，表示主要使用带有显式类别标签的数据集（如 ImageNet 或 OpenImages）学习。对于视觉-语言，流行的数据集如 Conceptual Captions、MSCOCO 或 CLIP 都涉及非平凡的数据收集（和清理）过程。这种昂贵的策划过程限制了数据集的大小，从而阻碍了训练模型的扩展。在本文中，我们利用了一个超过十亿图像替代文本对的噪声数据集，该数据集无需 Conceptual Captions 数据集中昂贵的过滤或后处理步骤。一个简单的双编码器架构使用对比损失学习对齐图像和文本对的视觉与语言表示。我们表明，语料库的规模可以弥补其噪声，即使使用如此简单的学习方案也能产生最先进的表示。我们的视觉表示在转移到 ImageNet 和 VTAB 等分类任务时表现出色。对齐的视觉和语言表示实现了零样本图像分类，并在 Flickr30K 和 MSCOCO 图像-文本检索基准上创造了新的最先进结果，即使与更复杂的交叉注意力模型相比也是如此。这些表示还支持复杂文本和文本+图像查询的跨模态搜索。

Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.

核心贡献 · Key contributions

• 利用超过十亿图像 alt-text 对的噪声数据集，无需昂贵过滤。
  Leverages a noisy dataset of over one billion image alt-text pairs without expensive filtering.
• 采用简单双编码器架构和对比损失对齐视觉与语言表示。
  Simple dual-encoder architecture with contrastive loss aligns visual and language representations.
• 在 ImageNet 上实现 76.4% top-1 准确率的零样本图像分类。
  Achieves state-of-the-art zero-shot image classification on ImageNet with 76.4% top-1 accuracy.
• 在 Flickr30K 和 MSCOCO 图像-文本检索基准上创下新纪录。
  Sets new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks.
• 支持复杂文本和文本+图像查询的跨模态搜索。
  Enables cross-modality search with complex text and text + image queries.
• 表明数据规模可弥补噪声，以 4 倍大小超越精心策划的数据集。
  Shows that data scale can compensate for noise, outperforming curated datasets with 4x size.

局限 · Limitations

• 模型可能强化来自噪声网络 alt-text 数据的有害刻板印象。
  Model may reinforce harmful stereotypes from noisy web alt-text data.
• 需要数据平衡以防止跨人口群体的性能偏差。
  Data balancing needed to prevent skewed performance across demographic groups.
• 应禁止用于监控或恶意目的的潜在滥用。
  Potential misuse for surveillance or nefarious purposes should be prohibited.
• 需要对敏感宗教或文化物品进行额外测试以减轻错误标签影响。
  Additional testing required for sensitive religious or cultural items to mitigate mislabeling.
• 在需要超越检索的细粒度视觉推理任务上，模型性能可能下降。
  Model performance may degrade on tasks requiring fine-grained visual reasoning beyond retrieval.

论文章节 · Sections（共 19）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2 相关工作 2 Related Work
• 3 大规模含噪图文数据集 3 A Large-Scale Noisy Image-Text Dataset
• 4.1 含噪图文对预训练 4.1 Pre-training on Noisy Image-Text Pairs
• 4.2 迁移至图文匹配与检索 4.2 Transferring to Image-Text Matching & Retrieval
• 4.3 迁移至视觉分类 4.3 Transferring to Visual Classification
• 5 实验与结果 5 Experiments and Results
• 5.1 图文匹配与检索 5.1 Image-Text Matching & Retrieval
• 5.2 零样本视觉分类 5.2 Zero-shot Visual Classification
• 5.3 仅图像编码器的视觉分类 5.3 Visual Classification w/ Image Encoder Only
• 6 消融研究 6 Ablation Study
• 6.1 模型架构 6.1 Model Architectures
• 6.2 预训练数据集 6.2 Pre-training Datasets
• 7 学习嵌入分析 7 Analysis of Learned Embeddings
• 8 多语言 ALIGN 模型 8 Multilingual ALIGN Model
• 9 结论 9 Conclusion
• 10 社会影响与未来工作 10 Social Impacts and Future Work
• 致谢 Acknowledgements

阅读逐段中英对照全文 →