Unifying multimodal understanding and generation has shown impressive capabilities in cutting-edge proprietary systems. In this work, we introduce BAGEL, an open-source foundational model that natively supports multimodal understanding and generation. BAGEL is a unified, decoder-only model pretrained on trillions of tokens curated from large-scale interleaved text, image, video, and web data. When scaled with such diverse multimodal interleaved data, BAGEL exhibits emerging capabilities in complex multimodal reasoning. As a result, it significantly outperforms open-source unified models in both multimodal generation and understanding across standard benchmarks, while exhibiting advanced multimodal reasoning abilities such as free-form image manipulation, future frame prediction, 3D manipulation, and world navigation. In the hope of facilitating further opportunities for multimodal research, we share the key findings, pretraining details, data creation protocal, and release our code and checkpoints to the community. The project page is at https://bagel-ai.org/
核心贡献 · Key contributions
提出 BAGEL，一个开源统一多模态基础模型，同时支持理解与生成。 Introduces BAGEL, an open-source unified multimodal foundation model supporting both understanding and generation.
提出混合 Transformer（MoT）架构，通过共享自注意力实现无瓶颈的多模态交互。 Proposes a Mixture-of-Transformers (MoT) architecture with shared self-attention for bottleneck-free multimodal interaction.
开发了从网络和视频源构建高质量多模态交错数据的可扩展协议。 Develops a scalable protocol for constructing high-quality multimodal interleaved data from web and video sources.
展示了复杂多模态推理中的涌现能力，包括自由形式图像操作和世界导航。 Demonstrates emergent capabilities in complex multimodal reasoning, including free-form image manipulation and world navigation.
在标准多模态理解与生成基准上优于开源统一模型。 Outperforms open-source unified models on standard benchmarks for multimodal understanding and generation.
发布代码、检查点和训练细节以促进多模态研究。 Releases code, checkpoints, and training details to facilitate multimodal research.
局限 · Limitations
评估依赖的基准可能无法完全捕捉涌现能力。 Evaluation relies on benchmarks that may not fully capture emergent capabilities.
模型在 IP 生成和人体姿态等复杂任务上的性能仍有限。 Model performance on complex tasks like IP generation and human pose remains limited.
训练数据质量和多样性可能引入偏差或覆盖缺口。 Training data quality and diversity may introduce biases or coverage gaps.
更大模型规模和数据规模的可扩展性尚未充分探索。 Scalability to larger model sizes and data scales is not fully explored.