Aria：一个开放的多模态原生混合专家模型

Aria: An Open Multimodal Native Mixture-of-Experts Model

李俊男 Junnan Li · Rhymes AI · 2024-10-08 · arXiv:2410.05993 ↗ · 被引 139

摘要 · Abstract

信息以多种模态呈现。多模态原生 AI 模型对于整合现实世界信息并提供全面理解至关重要。尽管存在专有的多模态原生模型，但它们缺乏开放性，这给采用带来了障碍，更不用说适应了。为了填补这一空白，我们引入了 Aria，一个开放的多模态原生模型，在广泛的多模态、语言和编码任务中具有最佳性能。Aria 是一个混合专家模型，每个视觉令牌和文本令牌分别有 3.9B 和 3.5B 激活参数。它在各种多模态任务上优于 Pixtral-12B 和 Llama3.2-11B，并与最佳专有模型竞争。我们按照 4 阶段流程从头开始预训练 Aria，逐步使模型具备语言理解、多模态理解、长上下文窗口和指令遵循的强大能力。我们开源了模型权重以及一个代码库，以促进 Aria 在实际应用中的轻松采用和适应。

Information comes in diverse modalities. Multimodal native AI models are essential to integrate real-world information and deliver comprehensive understanding. While proprietary multimodal native models exist, their lack of openness imposes obstacles for adoptions, let alone adaptations. To fill this gap, we introduce Aria, an open multimodal native model with best-in-class performance across a wide range of multimodal, language, and coding tasks. Aria is a mixture-of-expert model with 3.9B and 3.5B activated parameters per visual token and text token, respectively. It outperforms Pixtral-12B and Llama3.2-11B, and is competitive against the best proprietary models on various multimodal tasks. We pre-train Aria from scratch following a 4-stage pipeline, which progressively equips the model with strong capabilities in language understanding, multimodal understanding, long context window, and instruction following. We open-source the model weights along with a codebase that facilitates easy adoptions and adaptations of Aria in real-world applications.

核心贡献 · Key contributions

提出 Aria，首个开源多模态原生 MoE 模型，性能领先同类。
Introduces Aria, the first open multimodal native MoE model with best-in-class performance.
提出四阶段训练流程，从头开发多模态原生模型。
Proposes a 4-stage training pipeline for developing multimodal native models from scratch.
在多模态、语言和编码任务上优于 Pixtral-12B 和 Llama3.2-11B。
Achieves superior performance over Pixtral-12B and Llama3.2-11B on multimodal, language, and coding tasks.
在多项多模态基准上与 GPT-4o 和 Gemini-1.5 等专有模型竞争。
Competes with proprietary models like GPT-4o and Gemini-1.5 on various multimodal benchmarks.
开源模型权重和训练框架，便于采用和微调。
Open-sources model weights and a training framework for easy adoption and fine-tuning.
展示出强大的长上下文多模态理解和指令遵循能力。
Demonstrates strong long-context multimodal understanding and instruction following.

局限 · Limitations

模型性能在特定基准上评估，实际泛化能力可能不同。
Model performance is evaluated on specific benchmarks; real-world generalization may vary.
64K 上下文窗口可能不足以处理极长的多模态序列。
The 64K context window may not suffice for extremely long multimodal sequences.
训练数据截至 2024 年 5 月，未涵盖更新信息。
Training data is limited to knowledge until May 2024; newer information is not covered.
视觉编码器基于 SigLIP-SO400M 预训练，其他视觉领域效果可能较差。
The model's visual encoder is pretrained on SigLIP-SO400M; other visual domains may be less effective.
专家专业化分析限于三个领域，更广泛的泛化未确认。
Expert specialization analysis is limited to three domains; broader generalization not confirmed.

论文章节 · Sections（共 13）

摘要 Abstract
1 引言 1 Introduction
2.1 细粒度混合专家 2.1 Fine-Grained Mixture-of-Experts
2.2 视觉编码器 2.2 Visual Encoder
2.3 基础设施 2.3 Infrastructure
3 训练 3 Training
3.1 语言预训练 3.1 Language Pre-training
3.2 多模态预训练 3.2 Multimodal Pre-training
3.3 多模态长上下文预训练 3.3 Multimodal Long-Context Pre-training
3.4 多模态后训练 3.4 Multimodal Post-training
4.1 基准测试结果 4.1 Benchmark Results
4.2 专家模态特化 4.2 Expert Modality Specialization
5 致谢 5 Acknowledgement

阅读逐段中英对照全文 →