Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner. Code, models, and datasets are released at https://github.com/salesforce/BLIP.
核心贡献 · Key contributions
提出 BLIP，一个统一的视觉语言预训练框架，同时适用于理解和生成任务。 Proposes BLIP, a unified VLP framework for both understanding and generation tasks.
引入多模态混合编码器-解码器（MED），实现灵活的多任务预训练。 Introduces multimodal mixture of encoder-decoder (MED) for flexible multi-task pre-training.
开发 CapFilt 方法，通过生成合成描述和过滤噪声来引导网络数据。 Develops CapFilt to bootstrap noisy web data by generating synthetic captions and filtering noise.
在图像-文本检索、图像描述、视觉问答和零样本视频任务上取得最先进结果。 Achieves state-of-the-art results on image-text retrieval, captioning, VQA, and zero-shot video tasks.
通过零样本迁移展示了在视频-语言任务上的强泛化能力。 Demonstrates strong generalization to video-language tasks via zero-shot transfer.
发布代码、模型和引导数据集以促进未来研究。 Releases code, models, and bootstrapped datasets to facilitate future research.
局限 · Limitations
CapFilt 需要在人工标注数据（如 COCO）上微调描述生成器和过滤器。 CapFilt requires finetuning captioner and filter on human-annotated data like COCO.
使用确定性解码（如束搜索）时，CapFilt 的性能提升会减弱。 Performance gains from CapFilt diminish when using deterministic decoding like beam search.
零样本视频迁移忽略时间信息，限制了在时间任务上的性能。 Zero-shot video transfer ignores temporal information, limiting performance on temporal tasks.
引导数据集需要从头训练新模型；从教师模型继续训练无效。 Bootstrapped dataset requires training a new model from scratch; continued training from teacher is ineffective.
方法依赖大规模网络数据和计算资源进行预训练。 Method relies on large-scale web data and computational resources for pre-training.
论文章节 · Sections（共 24）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
2.1 视觉-语言预训练2.1 Vision-language Pre-training
2.2 知识蒸馏2.2 Knowledge Distillation
2.3 数据增强2.3 Data Augmentation
3 方法3 Method
3.1 模型架构3.1 Model Architecture
3.2 预训练目标3.2 Pre-training Objectives
3.3 CapFilt3.3 CapFilt
4 实验与讨论4 Experiments and Discussions
4.1 预训练细节4.1 Pre-training Details
4.2 CapFilt 的效果4.2 Effect of CapFilt
4.3 多样性对合成标题至关重要4.3 Diversity is Key for Synthetic Captions
4.4 参数共享与解耦4.4 Parameter Sharing and Decoupling
5 与最新技术的比较5 Comparison with State-of-the-arts
5.1 图像-文本检索5.1 Image-Text Retrieval
5.2 图像描述5.2 Image Captioning
5.3 视觉问答5.3 Visual Question Answering (VQA)
5.4 自然语言视觉推理5.4 Natural Language Visual Reasoning (NLVR2)
5.5 视觉对话5.5 Visual Dialog (VisDial)
5.6 零样本迁移到视频-语言任务5.6 Zero-shot Transfer to Video-Language Tasks