In this work, we introduce the Qwen-VL series, a set of large-scale vision-language models (LVLMs) designed to perceive and understand both texts and images. Starting from the Qwen-LM as a foundation, we endow it with visual capacity by the meticulously designed (i) visual receptor, (ii) input-output interface, (iii) 3-stage training pipeline, and (iv) multilingual multimodal cleaned corpus. Beyond the conventional image description and question-answering, we implement the grounding and text-reading ability of Qwen-VLs by aligning image-caption-box tuples. The resulting models, including Qwen-VL and Qwen-VL-Chat, set new records for generalist models under similar model scales on a broad range of visual-centric benchmarks (e.g., image captioning, question answering, visual grounding) and different settings (e.g., zero-shot, few-shot). Moreover, on real-world dialog benchmarks, our instruction-tuned Qwen-VL-Chat also demonstrates superiority compared to existing vision-language chatbots. Code, demo and models are available at https://github.com/QwenLM/Qwen-VL.
核心贡献 · Key contributions
提出 Qwen-VL 系列，大规模视觉语言模型，具备细粒度理解、定位和文本读取能力。 Introduced Qwen-VL series, large-scale vision-language models with fine-grained understanding, grounding, and text-reading.
设计了包含视觉受体、输入输出接口和多语言清洗语料的三阶段训练流程。 Designed a 3-stage training pipeline with a visual receptor, input-output interface, and multilingual cleaned corpus.
在图像描述、视觉问答、视觉定位和文本导向基准上取得最先进结果。 Achieved state-of-the-art results on image captioning, VQA, visual grounding, and text-oriented benchmarks.
展示了强大的少样本和零样本性能，与 Flamingo-80B 等更大模型相当。 Demonstrated strong few-shot and zero-shot performance, comparable to larger models like Flamingo-80B.
Qwen-VL-Chat 在 TouchStone 和 MME 等真实对话基准上优于现有聊天机器人。 Qwen-VL-Chat outperformed existing chatbots on real-world dialog benchmarks like TouchStone and MME.
发布开源模型，支持多语言、多图像和细粒度视觉理解。 Released open-source models supporting multilingual, multi-image, and fine-grained visual understanding.
局限 · Limitations
限于 7B 参数规模；更大模型可能获得更好性能。 Limited to 7B parameter scale; larger models may yield better performance.
视觉编码器分辨率上限为 448x448；更高分辨率可提升细粒度任务。 Visual encoder resolution capped at 448x448; higher resolutions could improve fine-grained tasks.
训练数据主要为英文和中文；其他语言性能可能受限。 Training data primarily English and Chinese; performance on other languages may be limited.
未集成语音或视频等其他模态；需要未来工作。 No integration with other modalities like speech or video; future work needed.
指令微调数据量（35 万）相对较小；更大数据集可增强对话能力。 Instruction tuning data size (350k) relatively small; larger datasets could enhance dialogue ability.
论文章节 · Sections（共 16）
摘要Abstract
引言Introduction
2.1 模型架构2.1 Model Architecture
2.2 输入与输出2.2 Inputs and Outputs
训练Training
3.1 预训练3.1 Pre-training
3.2 多任务预训练3.2 Multi-task Pre-training
3.3 有监督微调3.3 Supervised Fine-tuning
评估Evaluation
4.1 图像描述与通用视觉问答4.1 Image Caption and General Visual Question Answering