CogVLM：预训练语言模型的视觉专家

CogVLM: Visual Expert for Pretrained Language Models

唐杰 Jie Tang · Zhipu AI · 2023-11-06 · arXiv:2311.03079 ↗ · 被引 818

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

我们介绍了 CogVLM，一个强大的开源视觉语言基础模型。与流行的浅层对齐方法（将图像特征映射到语言模型的输入空间）不同，CogVLM 通过在注意力层和前馈网络层中引入可训练的视觉专家模块，弥合了冻结的预训练语言模型与图像编码器之间的差距。因此，CogVLM 实现了视觉语言特征的深度融合，同时不牺牲任何 NLP 任务的性能。CogVLM-17B 在 10 个经典跨模态基准上取得了最先进的结果，包括 NoCaps、Flicker30k 描述、RefCOCO、RefCOCO+、RefCOCOg、Visual7W、GQA、ScienceQA、VizWiz VQA 和 TDIUC，并在 VQAv2、OKVQA、TextVQA、COCO 描述等任务上排名第二，超越或媲美 PaLI-X 55B。代码和检查点可在 https://github.com/THUDM/CogVLM 获取。

We introduce CogVLM, a powerful open-source visual language foundation model. Different from the popular shallow alignment method which maps image features into the input space of language model, CogVLM bridges the gap between the frozen pretrained language model and image encoder by a trainable visual expert module in the attention and FFN layers. As a result, CogVLM enables deep fusion of vision language features without sacrificing any performance on NLP tasks. CogVLM-17B achieves state-of-the-art performance on 10 classic cross-modal benchmarks, including NoCaps, Flicker30k captioning, RefCOCO, RefCOCO+, RefCOCOg, Visual7W, GQA, ScienceQA, VizWiz VQA and TDIUC, and ranks the 2nd on VQAv2, OKVQA, TextVQA, COCO captioning, etc., surpassing or matching PaLI-X 55B. Codes and checkpoints are available at https://github.com/THUDM/CogVLM.

核心贡献 · Key contributions

• 提出 CogVLM，一种通过可训练视觉专家模块实现视觉与语言特征深度融合的视觉语言模型。
  Introduces CogVLM, a visual language model with a trainable visual expert module for deep fusion of vision and language features.
• 在 17 个跨模态基准上达到最先进水平，涵盖图像描述、视觉问答和视觉定位。
  Achieves state-of-the-art on 17 cross-modal benchmarks, including image captioning, VQA, and visual grounding.
• 通过冻结预训练语言模型参数并添加视觉专家，完整保留其 NLP 能力。
  Preserves full NLP capabilities of the pretrained language model by freezing its parameters and adding visual experts.
• 通过大量消融实验验证了视觉专家模块和深度融合的有效性。
  Validates the visual expert module and deep fusion through extensive ablation studies.
• 公开模型权重和 SFT 数据集，促进研究和工业应用。
  Releases model weights and SFT dataset to the public for research and industrial applications.
• 探索多模态预训练中的关键因素，包括视觉编码器规模、注意力掩码和自监督损失。
  Explores critical factors in multimodal pretraining, including visual encoder scale, attention mask, and self-supervised loss.

局限 · Limitations

• CogVLM-17B 基于 Vicuna-7B 训练，限制了向更大语言模型的扩展性。
  CogVLM-17B is trained from Vicuna-7B, limiting scalability to larger language models.
• 视觉专家模块使参数量翻倍，增加了内存和存储需求。
  The visual expert module doubles parameters, increasing memory and storage requirements.
• 在 TextVQA 等文本密集型任务上，性能依赖视觉编码器规模，较小编码器会导致下降。
  Performance on text-heavy tasks like TextVQA relies on visual encoder scale, with smaller encoders causing degradation.
• 模型需要大规模预训练数据（15 亿图文对），可能不适用于所有领域。
  The model requires large-scale pretraining data (1.5B image-text pairs), which may not be available for all domains.
• 消融实验仅进行 6000 次迭代，可能无法反映完全收敛的行为。
  Ablation studies are limited to 6,000 iterations, which may not reflect full convergence behavior.

论文章节 · Sections（共 11）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2.1 架构 2.1 Architecture
• 2.2 预训练 2.2 Pretraining
• 2.3 对齐 2.3 Alignment
• 3 实验 3 Experiments
• 3.1 图像描述 3.1 Image Captioning
• 3.2 视觉问答 3.2 Visual Question Answering
• 3.3 LVLM 基准测试 3.3 LVLM Benchmarks
• 3.5 消融研究 3.5 Ablation Study
• 4 结论 4 Conclusion

阅读逐段中英对照全文 →