Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this note, we show that the fully-connected vision-language cross-modal connector in LLaVA is surprisingly powerful and data-efficient. With simple modifications to LLaVA, namely, using CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with simple response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in ~1 day on a single 8-A100 node. We hope this can make state-of-the-art LMM research more accessible. Code and model will be publicly available.
核心贡献 · Key contributions
展示了 LLaVA 的全连接视觉-语言连接器强大且数据高效。 Shows that LLaVA's fully-connected vision-language connector is powerful and data-efficient.
引入简单改进：MLP 投影和带响应格式提示的学术 VQA 数据。 Introduces simple modifications: MLP projection and academic VQA data with response formatting prompts.
仅用 120 万公开数据和 8 块 A100 约 1 天训练，在 11 个基准上达到最先进水平。 Achieves state-of-the-art across 11 benchmarks with only 1.2M public data and ~1 day training on 8 A100s.
证明视觉指令微调比大规模预训练对 LMM 更重要。 Demonstrates that visual instruction tuning is more important than large-scale pre-training for LMMs.
提出单一响应格式提示，平衡短和长形式答案，无需额外数据处理。 Proposes a single response formatting prompt to balance short and long-form answers without extra data processing.
为未来 LMM 研究提供完全可复现且负担得起的基线。 Provides fully reproducible and affordable baselines for future LMM research.
局限 · Limitations
使用完整图像块，相比视觉重采样器延长训练迭代。 Uses full image patches, prolonging training iterations compared to visual resamplers.
由于缺乏数据和上下文长度限制，无法处理多张图像。 Cannot process multiple images due to lack of data and context length limits.
尽管能遵循复杂指令，但在某些领域问题解决能力有限。 Problem-solving capabilities limited in certain domains despite complex instruction following.
无法避免幻觉和错误信息，在关键应用中需谨慎使用。 Not exempt from hallucinations and misinformation, requiring caution in critical applications.
论文章节 · Sections（共 17）
摘要Abstract
1 引言1 Introduction
指令跟随型大语言模型Instruction-following LMM.
多模态指令跟随数据Multimodal instruction-following data.
概述Overview.
响应格式提示Response formatting prompts.
MLP 视觉-语言连接器MLP vision-language connector.
学术任务导向数据Academic task oriented data.
额外扩展Additional scaling.
与最先进技术比较Comparison with SoTA.
零样本格式指令泛化Zero-shot format instruction generalization.