MiniGPT-4：利用先进大语言模型增强视觉语言理解

MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

朱德耀 Deyao Zhu · KAUST · 2023-04-20 · arXiv:2304.10592 ↗ · 被引 3208

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

最近的 GPT-4 展示了非凡的多模态能力，例如从手写文本直接生成网站以及识别图像中的幽默元素。这些特征在以往的视觉语言模型中很少见。然而，GPT-4 背后的技术细节仍未公开。我们认为 GPT-4 增强的多模态生成能力源于对先进大语言模型（LLM）的利用。为了研究这一现象，我们提出了 MiniGPT-4，它使用一个投影层将冻结的视觉编码器与冻结的先进 LLM Vicuna 对齐。我们的工作首次揭示，将视觉特征与先进大语言模型适当对齐可以拥有 GPT-4 展示的许多高级多模态能力，例如生成详细的图像描述和根据手绘草稿创建网站。此外，我们还观察到 MiniGPT-4 的其他新兴能力，包括根据给定图像编写故事和诗歌，根据食物照片教用户烹饪等。在我们的实验中，我们发现仅在短图像描述对上训练的模型会产生不自然的语言输出（例如重复和碎片化）。为了解决这个问题，我们在第二阶段整理了一个详细的图像描述数据集来微调模型，从而提高了模型生成的可靠性和整体可用性。我们的代码、预训练模型和收集的数据集可在 https://minigpt-4.github.io/获取。

The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text and identifying humorous elements within images. These features are rarely observed in previous vision-language models. However, the technical details behind GPT-4 continue to remain undisclosed. We believe that the enhanced multi-modal generation capabilities of GPT-4 stem from the utilization of sophisticated large language models (LLM). To examine this phenomenon, we present MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer. Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, such as detailed image description generation and website creation from hand-drawn drafts. Furthermore, we also observe other emerging capabilities in MiniGPT-4, including writing stories and poems inspired by given images, teaching users how to cook based on food photos, and so on. In our experiment, we found that the model trained on short image caption pairs could produce unnatural language outputs (e.g., repetition and fragmentation). To address this problem, we curate a detailed image description dataset in the second stage to finetune the model, which consequently improves the model's generation reliability and overall usability. Our code, pre-trained model, and collected dataset are available at https://minigpt-4.github.io/.

核心贡献 · Key contributions

• 提出 MiniGPT-4，通过一个投影层将冻结的视觉编码器与 Vicuna 对齐。
  Proposes MiniGPT-4, aligning a frozen visual encoder with Vicuna via one projection layer.
• 揭示适当的视觉-LLM 对齐可产生类似 GPT-4 的能力，如详细图像描述和网站创建。
  Reveals that proper visual-LLM alignment yields GPT-4-like abilities like detailed image description and website creation.
• 发现短标题预训练导致不自然输出；第二阶段使用详细描述微调提高了可靠性。
  Identifies that short caption pretraining causes unnatural outputs; second-stage finetuning with detailed descriptions improves reliability.
• 展示 MiniGPT-4 的新兴能力，包括写诗、生成食谱和解读梗图。
  Demonstrates emergent abilities in MiniGPT-4, including poem writing, recipe generation, and meme interpretation.
• 表明在有限数据设置下，单层线性投影足以实现视觉-语言对齐。
  Shows that a single linear projection layer suffices for vision-language alignment in limited data settings.
• 提供开源代码、预训练模型和精选数据集以促进可复现性。
  Provides open-source code, pretrained model, and curated dataset for reproducibility.

局限 · Limitations

• MiniGPT-4 继承了 LLM 的幻觉问题，例如错误描述不存在的物体。
  MiniGPT-4 inherits LLM hallucination, e.g., falsely describing nonexistent objects.
• 视觉感知有限，尤其在空间定位任务中。
  Visual perception is limited, especially in spatial localization tasks.
• 第二阶段数据集较小（3500 对）且人工筛选，限制了可扩展性。
  Second-stage dataset is small (3,500 pairs) and manually curated, limiting scalability.
• 模型依赖冻结组件；未对视觉编码器或 LLM 进行端到端训练。
  Model relies on frozen components; no end-to-end training of vision encoder or LLM.
• 高级任务的评估是定性的，缺乏标准化基准。
  Evaluation on advanced tasks is qualitative and lacks standardized benchmarks.

论文章节 · Sections（共 14）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2 相关工作 2 Related Works
• 3 方法 3 Method
• 3.1 第一预训练阶段 3.1 First pretraining stage
• 3.2 构建视觉语言领域的高质量对齐数据集 3.2 Curating a high-quality alignment dataset for vision-language domain.
• 3.3 第二阶段微调 3.3 Second-stage finetuning
• 4 实验 4 Experiments
• 4.1 通过定性示例揭示 MiniGPT-4 的涌现能力 4.1 Uncovering emergent abilities with MiniGPT-4 through qualitative examples
• 4.2 定量分析 4.2 Quantitative analysis
• 4.3 第二阶段微调分析 4.3 Analysis on the second-stage finetuning
• 4.4 架构设计消融实验 4.4 Ablation on the architecture designs
• 4.5 局限性分析 4.5 Limitation analysis
• 5 讨论 5 Discussion

阅读逐段中英对照全文 →