The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.
核心贡献 · Key contributions
提出 BLIP-2，一种通用且高效的视觉-语言预训练策略，利用冻结的图像编码器和 LLM。 Proposes BLIP-2, a generic and efficient vision-language pre-training strategy using frozen image encoders and LLMs.
引入轻量级 Querying Transformer（Q-Former），通过两阶段预训练弥合模态差距。 Introduces a lightweight Querying Transformer (Q-Former) to bridge the modality gap with two-stage pre-training.
在 VQA、图像描述和检索任务上达到最先进性能，且可训练参数显著减少。 Achieves state-of-the-art performance on VQA, image captioning, and retrieval with significantly fewer trainable parameters.
展示了遵循自然语言指令的零样本图像到文本生成，支持视觉推理和对话。 Demonstrates zero-shot image-to-text generation following natural language instructions, enabling visual reasoning and conversation.
验证了更强的冻结单模态模型（图像编码器或 LLM）能持续提升 VLP 性能。 Validates that stronger frozen unimodal models (image encoder or LLM) consistently improve VLP performance.
表明表示学习阶段对于防止生成学习期间冻结 LLM 的灾难性遗忘至关重要。 Shows representation learning stage is crucial to prevent catastrophic forgetting in frozen LLMs during generative learning.
局限 · Limitations
BLIP-2 缺乏上下文学习能力，因为预训练数据中每个样本仅包含单个图像-文本对。 BLIP-2 lacks in-context learning capability due to single image-text pair per sample in pre-training data.
图像到文本生成可能因知识不准确、推理错误或信息过时而效果不佳。 Image-to-text generation may suffer from inaccurate knowledge, incorrect reasoning, or outdated information.
继承了冻结 LLM 的风险，包括冒犯性语言、社会偏见和隐私信息泄露。 Inherits risks from frozen LLMs, including offensive language, social bias, and private information leakage.
在 OK-VQA 等开放世界知识任务上的性能相比更大 LLM 有限。 Performance on open-world knowledge tasks like OK-VQA is limited compared to larger LLMs.
零样本任务需要精心设计提示；最佳提示因 LLM 类型而异。 Requires careful prompt engineering for zero-shot tasks; optimal prompts vary across LLM types.