Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.
核心贡献 · Key contributions
首次使用仅语言的 GPT-4 生成多模态语言-图像指令遵循数据。 First to use language-only GPT-4 to generate multimodal language-image instruction-following data.
提出 LLaVA，一种端到端训练的大型多模态模型，连接视觉编码器和 LLM。 Introduces LLaVA, an end-to-end trained large multimodal model connecting a vision encoder and LLM.
在合成多模态指令遵循数据集上达到 GPT-4 的 85.1%相对分数。 Achieves 85.1% relative score vs. GPT-4 on synthetic multimodal instruction-following dataset.
与 GPT-4 集成后在 Science QA 上达到 92.53%的最新准确率。 New state-of-the-art accuracy of 92.53% on Science QA when ensembled with GPT-4.
构建了两个用于视觉指令遵循的评估基准，包含多样化任务。 Constructs two evaluation benchmarks for visual instruction following with diverse tasks.
公开了生成的多模态指令数据、模型和代码。 Releases generated multimodal instruction data, model, and code publicly.
局限 · Limitations
数据生成依赖 GPT-4，可能引入偏差或不准确。 Data generation relies on GPT-4, which may introduce biases or inaccuracies.
LLaVA 有时在复杂语义上失败，例如将图像误解为补丁包。 LLaVA sometimes fails on complex semantics, e.g., misinterpreting image as bag of patches.
仅使用 158K 指令样本；更多数据可能提升性能。 Limited to 158K instruction samples; performance may improve with more data.
简单的线性投影用于视觉-语言连接可能限制表达能力。 Simple linear projection for vision-language connection may limit expressiveness.
评估基准是合成的或范围有限；未充分测试真实世界泛化。 Evaluation benchmarks are synthetic or limited in scope; real-world generalization not fully tested.
论文章节 · Sections（共 10）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 GPT 辅助的视觉指令数据生成3 GPT-assisted Visual Instruction Data Generation