CoCa: Contrastive Captioners are Image-Text Foundation Models
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→探索大规模预训练基础模型在计算机视觉中具有重要意义，因为这些模型可以快速迁移到许多下游任务。本文提出了对比字幕器（CoCa），一种极简设计，通过对比损失和字幕损失联合预训练图像-文本编码器-解码器基础模型，从而融合了 CLIP 等对比方法和 SimVLM 等生成方法的模型能力。与标准编码器-解码器变换器（所有解码器层都关注编码器输出）不同，CoCa 在前半部分解码器层中省略了交叉注意力以编码单模态文本表示，并级联剩余的解码器层，这些层交叉关注图像编码器以获取多模态图像-文本表示。我们在单模态图像和文本嵌入之间应用对比损失，此外在多模态解码器输出上应用字幕损失，以自回归方式预测文本标记。通过共享相同的计算图，两个训练目标以最小开销高效计算。CoCa 在网页规模的替代文本数据和标注图像上从头到尾进行预训练，将所有标签简单地视为文本，无缝统一自然语言监督以进行表示学习。实验表明，CoCa 在广泛的下游任务上通过零样本迁移或最小任务特定适应实现了最先进的性能，涵盖视觉识别（ImageNet、Kinetics-400/600/700、Moments-in-Time）、跨模态检索（MSCOCO、Flickr30K、MSR-VTT）、多模态理解（VQA、SNLI-VE、NLVR2）和图像字幕（MSCOCO、NoCaps）。值得注意的是，在 ImageNet 分类上，CoCa 获得了 86.3%的零样本 top-1 准确率，使用冻结编码器和学习分类头时达到 90.6%，以及使用微调编码器时达到新的最先进 91.0% top-1 准确率。
Exploring large-scale pretrained foundation models is of significant interest in computer vision because these models can be quickly transferred to many downstream tasks. This paper presents Contrastive Captioner (CoCa), a minimalist design to pretrain an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss, thereby subsuming model capabilities from contrastive approaches like CLIP and generative methods like SimVLM. In contrast to standard encoder-decoder transformers where all decoder layers attend to encoder outputs, CoCa omits cross-attention in the first half of decoder layers to encode unimodal text representations, and cascades the remaining decoder layers which cross-attend to the image encoder for multimodal image-text representations. We apply a contrastive loss between unimodal image and text embeddings, in addition to a captioning loss on the multimodal decoder outputs which predicts text tokens autoregressively. By sharing the same computational graph, the two training objectives are computed efficiently with minimal overhead. CoCa is pretrained end-to-end and from scratch on both web-scale alt-text data and annotated images by treating all labels simply as text, seamlessly unifying natural language supervision for representation learning. Empirically, CoCa achieves state-of-the-art performance with zero-shot transfer or minimal task-specific adaptation on a broad range of downstream tasks, spanning visual recognition (ImageNet, Kinetics-400/600/700, Moments-in-Time), crossmodal retrieval (MSCOCO, Flickr30K, MSR-VTT), multimodal understanding (VQA, SNLI-VE, NLVR2), and image captioning (MSCOCO, NoCaps). Notably on ImageNet classification, CoCa obtains 86.3% zero-shot top-1 accuracy, 90.6% with a frozen encoder and learned classification head, and new state-of-the-art 91.0% top-1 accuracy on ImageNet with a finetuned encoder.