We present the Qwen2-VL Series, an advanced upgrade of the previous Qwen-VL models that redefines the conventional predetermined-resolution approach in visual processing. Qwen2-VL introduces the Naive Dynamic Resolution mechanism, which enables the model to dynamically process images of varying resolutions into different numbers of visual tokens. This approach allows the model to generate more efficient and accurate visual representations, closely aligning with human perceptual processes. The model also integrates Multimodal Rotary Position Embedding (M-RoPE), facilitating the effective fusion of positional information across text, images, and videos. We employ a unified paradigm for processing both images and videos, enhancing the model's visual perception capabilities. To explore the potential of large multimodal models, Qwen2-VL investigates the scaling laws for large vision-language models (LVLMs). By scaling both the model size-with versions at 2B, 8B, and 72B parameters-and the amount of training data, the Qwen2-VL Series achieves highly competitive performance. Notably, the Qwen2-VL-72B model achieves results comparable to leading models such as GPT-4o and Claude3.5-Sonnet across various multimodal benchmarks, outperforming other generalist models. Code is available at https://github.com/QwenLM/Qwen2-VL .
核心贡献 · Key contributions
提出朴素动态分辨率机制，可将任意分辨率图像处理为可变数量的视觉标记。 Introduces Naive Dynamic Resolution mechanism for processing images at any resolution into variable visual tokens.
提出多模态旋转位置嵌入（M-RoPE），融合文本、图像和视频的位置信息。 Proposes Multimodal Rotary Position Embedding (M-RoPE) to fuse positional info across text, images, and videos.
统一图像和视频处理范式，通过 3D 卷积和帧采样增强视觉感知。 Unifies image and video processing paradigm, enhancing visual perception with 3D convolutions and frame sampling.
探索大型视觉语言模型的缩放定律，在 2B、8B 和 72B 规模上取得有竞争力的性能。 Explores scaling laws for large vision-language models, achieving competitive performance at 2B, 8B, and 72B scales.
在多个基准上取得最先进结果，与 GPT-4o 和 Claude3.5-Sonnet 相当。 Achieves state-of-the-art results on multiple benchmarks, comparable to GPT-4o and Claude3.5-Sonnet.
支持图像中多语言文本理解，并具备设备操作的强大智能体能力。 Supports multilingual text understanding in images and robust agent capabilities for device operation.
局限 · Limitations
在 MMMU 基准上仍落后于 GPT-4o，表明复杂推理能力有待提升。 Performance on MMMU benchmark still lags behind GPT-4o, indicating room for improvement in complex reasoning.
评估时每视频最多提取 768 帧，影响长视频性能。 Video understanding limited to 768 frames per video during evaluation, affecting long video performance.
视觉语言导航性能远落后于专用模型，因 3D 地图建模不完整。 Vision-language navigation performance significantly behind specialized models due to incomplete 3D map modeling.
过度放大微小图像可能导致分布外样本和性能下降。 Excessive upscaling of small images can lead to out-of-distribution samples and performance decline.
训练数据截止于 2023 年 6 月，限制了对近期事件的了解。 Training data cutoff date is June 2023, limiting knowledge of recent events.