Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-language models is challenging due to the rich input distributions and task diversity resulting from the additional visual input. Although vision-language pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models. We gather 26 publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA questions with image contexts). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-sourced at https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.
核心贡献 · Key contributions
首次对视觉语言指令微调进行系统研究，将 26 个数据集转化为指令格式。 First comprehensive study on vision-language instruction tuning with 26 datasets transformed into instruction format.
提出指令感知的查询变换器，用于提取任务相关的视觉特征。 Proposes instruction-aware Query Transformer for extracting task-relevant visual features.
在 13 个保留数据集上取得最先进的零样本性能，优于 BLIP-2 和 Flamingo。 Achieves state-of-the-art zero-shot performance on 13 held-out datasets, outperforming BLIP-2 and Flamingo.
引入平衡采样策略，以同步不同数据集的学习进度。 Introduces balanced sampling strategy to synchronize learning across diverse datasets.
在下游任务（如 ScienceQA 90.7%准确率）上展示最先进的微调结果。 Demonstrates state-of-the-art finetuning results on downstream tasks like ScienceQA (90.7% accuracy).
开源所有 InstructBLIP 模型，以促进可重复性和进一步研究。 Open-sources all InstructBLIP models for reproducibility and further research.
局限 · Limitations
依赖冻结的图像编码器和 LLM，限制了对新视觉领域的适应。 Relies on frozen image encoder and LLM, limiting adaptation to new visual domains.
指令微调数据限于 26 个数据集，可能无法覆盖所有真实世界任务。 Instruction tuning data limited to 26 datasets; may not cover all real-world tasks.
在完全未见过的任务类别（如视频问答）上的零样本性能仍落后于监督方法。 Zero-shot performance on entirely unseen task categories (e.g., video QA) still lags behind supervised methods.
需要仔细手动调整采样权重以达到最佳性能。 Requires careful manual adjustment of sampling weights for optimal performance.
定性评估限于少数示例，需要更广泛的用户研究。 Qualitative evaluation limited to a few examples; broader user studies needed.