The human ability to easily solve multimodal tasks in context (i.e., with only a few demonstrations or simple instructions), is what current multimodal systems have largely struggled to imitate. In this work, we demonstrate that the task-agnostic in-context learning capabilities of large multimodal models can be significantly enhanced by effective scaling-up. We introduce Emu2, a generative multimodal model with 37 billion parameters, trained on large-scale multimodal sequences with a unified autoregressive objective. Emu2 exhibits strong multimodal in-context learning abilities, even emerging to solve tasks that require on-the-fly reasoning, such as visual prompting and object-grounded generation. The model sets a new record on multiple multimodal understanding tasks in few-shot settings. When instruction-tuned to follow specific instructions, Emu2 further achieves new state-of-the-art on challenging tasks such as question answering benchmarks for large multimodal models and open-ended subject-driven generation. These achievements demonstrate that Emu2 can serve as a base model and general-purpose interface for a wide range of multimodal tasks. Code and models are publicly available to facilitate future research.
核心贡献 · Key contributions
提出 Emu2，一个 370 亿参数的生成式多模态模型，通过统一的自回归目标在大规模多模态序列上训练。 Introduces Emu2, a 37B parameter generative multimodal model trained with a unified autoregressive objective on large-scale multimodal sequences.
证明规模扩张能显著增强生成式多模态模型的任务无关上下文学习能力，包括视觉提示和对象锚定生成。 Demonstrates that scaling up generative multimodal models significantly enhances task-agnostic in-context learning abilities, including visual prompting and object-grounded generation.
在多个多模态理解基准（如 VQAv2 和 OKVQA）上取得少样本最优性能。 Achieves state-of-the-art few-shot performance on multiple multimodal understanding benchmarks, such as VQAv2 and OKVQA.
经指令微调后，Emu2-Chat 在 MM-Vet 和 TouchStone 等 LMM 基准上创下新纪录，超越此前公开模型。 After instruction tuning, Emu2-Chat sets new records on LMM benchmarks like MM-Vet and TouchStone, surpassing prior public models.
Emu2-Gen 支持上下文中的可控视觉生成，在主题驱动生成和多实体组合方面表现优异。 Emu2-Gen enables controllable visual generation in context, excelling in subject-driven generation and multi-entity composition.
公开代码和模型，以促进多模态上下文学习的未来研究。 Provides publicly available code and models to facilitate future research in multimodal in-context learning.
局限 · Limitations
上下文学习在复杂场景（如人群计数）中可能失败。 In-context learning may fail in complex scenes, e.g., counting in a crowd.
问答能力仍落后于 GPT-4V 等封闭多模态系统。 Question-answering capability still lags behind closed multimodal systems like GPT-4V.
由于训练数据中的偏见，可能生成有害或有偏见的内容。 Potential for generating harmful or biased content due to biases in training data.
模型幻觉可能导致某些情况下的错误预测。 Model hallucination may cause incorrect predictions in certain cases.
在需要细粒度推理或高分辨率理解的任务中仍存在性能差距。 Performance gap remains in tasks requiring fine-grained reasoning or high-resolution understanding.