The exponential growth of large language models (LLMs) has opened up numerous possibilities for multimodal AGI systems. However, the progress in vision and vision-language foundation models, which are also critical elements of multi-modal AGI, has not kept pace with LLMs. In this work, we design a large-scale vision-language foundation model (InternVL), which scales up the vision foundation model to 6 billion parameters and progressively aligns it with the LLM, using web-scale image-text data from various sources. This model can be broadly applied to and achieve state-of-the-art performance on 32 generic visual-linguistic benchmarks including visual perception tasks such as image-level or pixel-level recognition, vision-language tasks such as zero-shot image/video classification, zero-shot image/video-text retrieval, and link with LLMs to create multi-modal dialogue systems. It has powerful visual capabilities and can be a good alternative to the ViT-22B. We hope that our research could contribute to the development of multi-modal large models. Code and models are available at https://github.com/OpenGVLab/InternVL.
核心贡献 · Key contributions
提出 InternVL，一种大规模视觉语言基础模型，将视觉编码器扩展到 60 亿参数并与大语言模型对齐。 Proposes InternVL, a large-scale vision-language foundation model scaling vision encoder to 6B parameters and aligning with LLM.
引入渐进式图像文本对齐策略，利用网络规模噪声数据进行对比学习，细粒度数据进行生成学习，实现高效训练。 Introduces a progressive image-text alignment strategy for efficient training using web-scale noisy data for contrastive learning and fine-grained data for generative learning.
在 32 个通用视觉语言基准上取得最先进性能，包括图像分类、语义分割、视频分类、检索和字幕生成。 Achieves state-of-the-art performance on 32 generic visual-linguistic benchmarks including image classification, semantic segmentation, video classification, retrieval, and captioning.
展示强大的视觉感知能力，可作为 ViT-22B 在视觉任务中的优秀替代方案。 Demonstrates strong visual perception capabilities, serving as a good alternative to ViT-22B for visual tasks.
通过视觉编码器和语言中间件 QLLaMA 实现灵活模型组合，支持对比和生成任务。 Enables flexible model composition for contrastive and generative tasks via vision encoder and language middleware QLLaMA.
与多种大语言模型（如 LLaMA、Vicuna 和 InternLM）展现出良好的兼容性，适用于多模态对话。 Shows promising compatibility with various LLMs like LLaMA, Vicuna, and InternLM for multi-modal dialogue.
局限 · Limitations
训练需要大量计算资源（第一阶段 640 块 A100 GPU），限制了可访问性。 Training requires massive computational resources (640 A100 GPUs for stage 1), limiting accessibility.
性能严重依赖网络规模噪声数据质量；数据过滤至关重要但可能丢弃有用样本。 Performance heavily depends on web-scale noisy data quality; data filtering is crucial but may discard useful samples.
模型规模大（60 亿视觉编码器+80 亿中间件）导致推理延迟高，尤其在大图像上。 The model's large size (6B vision encoder + 8B middleware) leads to high inference latency, especially with larger images.
对齐策略是渐进式的，但仍可能无法完全弥合视觉和语言之间的模态差距。 Alignment strategy is progressive but still may not fully bridge the modality gap between vision and language.
评估主要集中在英语和中文；对其他语言的多语言能力探索较少。 Evaluation focuses on English and Chinese; multilingual capabilities on other languages are less explored.