We introduce InternVL3, a significant advancement in the InternVL series featuring a native multimodal pre-training paradigm. Rather than adapting a text-only large language model (LLM) into a multimodal large language model (MLLM) that supports visual inputs, InternVL3 jointly acquires multimodal and linguistic capabilities from both diverse multimodal data and pure-text corpora during a single pre-training stage. This unified training paradigm effectively addresses the complexities and alignment challenges commonly encountered in conventional post-hoc training pipelines for MLLMs. To further improve performance and scalability, InternVL3 incorporates variable visual position encoding (V2PE) to support extended multimodal contexts, employs advanced post-training techniques such as supervised fine-tuning (SFT) and mixed preference optimization (MPO), and adopts test-time scaling strategies alongside an optimized training infrastructure. Extensive empirical evaluations demonstrate that InternVL3 delivers superior performance across a wide range of multi-modal tasks. In particular, InternVL3-78B achieves a score of 72.2 on the MMMU benchmark, setting a new state-of-the-art among open-source MLLMs. Its capabilities remain highly competitive with leading proprietary models, including ChatGPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro, while also maintaining strong pure-language proficiency. In pursuit of open-science principles, we will publicly release both the training data and model weights to foster further research and development in next-generation MLLMs.
核心贡献 · Key contributions
提出原生多模态预训练，在单一阶段联合学习语言和多模态能力，避免事后对齐挑战。 Proposes native multimodal pre-training that jointly learns linguistic and multimodal capabilities in a single stage, avoiding post-hoc alignment challenges.
集成可变视觉位置编码（V2PE），通过灵活的位置增量支持扩展的多模态上下文。 Integrates Variable Visual Position Encoding (V2PE) to support extended multimodal contexts with flexible position increments.
采用监督微调和混合偏好优化（MPO）的高级后训练，增强推理和对齐。 Employs advanced post-training with supervised fine-tuning and mixed preference optimization (MPO) to enhance reasoning and alignment.
采用测试时缩放策略，结合视觉过程奖励模型（VisualPRM）进行最佳 N 选，提升推理性能。 Adopts test-time scaling strategies with a visual process reward model (VisualPRM) for best-of-N selection, improving reasoning performance.
在 MMMU 上达到开源多模态大语言模型中最优性能（72.2），与 GPT-4o 和 Gemini 2.5 Pro 等专有模型竞争。 Achieves state-of-the-art performance on MMMU (72.2) among open-source MLLMs, competitive with proprietary models like GPT-4o and Gemini 2.5 Pro.
公开释放训练数据和模型权重，促进下一代多模态大语言模型的开源研究。 Releases training data and model weights publicly to foster open-source research in next-generation MLLMs.
局限 · Limitations
在某些基准（如 MuirBench、HallusionBench）上仍落后于 Qwen2.5-VL 和 Gemini 2.5 Pro 等领先模型。 Performance on some benchmarks (e.g., MuirBench, HallusionBench) still lags behind leading models like Qwen2.5-VL and Gemini 2.5 Pro.
由于训练中缺乏额外的定位特定数据，视觉定位性能在大规模时趋于平稳。 Visual grounding performance plateaus at larger scales due to lack of additional grounding-specific data in training.
测试时缩放依赖独立的奖励模型（VisualPRM），增加了计算开销和复杂性。 Test-time scaling relies on a separate reward model (VisualPRM), adding computational overhead and complexity.
原生多模态预训练需要仔细平衡多模态和语言数据比例；最优比例可能不具泛化性。 Native multimodal pre-training requires careful balancing of multimodal and language data ratios; optimal ratio may not generalize.
评估主要在静态基准上进行；现实世界动态和交互场景（如 GUI 智能体）探索较少。 Evaluation primarily on static benchmarks; real-world dynamic and interactive scenarios (e.g., GUI agents) are less explored.
论文章节 · Sections（共 24）
摘要Abstract
1 引言1 Introduction
2 InternVL32 InternVL3
2.1 模型架构2.1 Model Architecture
2.2 原生多模态预训练2.2 Native Multimodal Pre-Training
2.3 后训练2.3 Post-Training
2.4 测试时扩展2.4 Test-Time Scaling
2.5 基础设施2.5 Infrastructure
3 实验3 Experiments
3.1 与其他先进多模态大模型的整体对比3.1 Overall Comparison to Other Advanced MLLMs
3.2 多模态推理与数学3.2 Multimodal Reasoning and Mathematics
3.3 OCR、图表与文档理解3.3 OCR, Chart, and Document Understanding