While next-token prediction is considered a promising path towards artificial general intelligence, it has struggled to excel in multimodal tasks, which are still dominated by diffusion models (e.g., Stable Diffusion) and compositional approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction. By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences. Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship models such as SDXL and LLaVA-1.6, while eliminating the need for diffusion or compositional architectures. Emu3 is also capable of generating high-fidelity video via predicting the next token in a video sequence. We simplify complex multimodal model designs by converging on a singular focus: tokens, unlocking great potential for scaling both during training and inference. Our results demonstrate that next-token prediction is a promising path towards building general multimodal intelligence beyond language. We open-source key techniques and models to support further research in this direction.
核心贡献 · Key contributions
Emu3 消除了扩散和组合架构，仅使用下一个词预测实现多模态生成与感知。 Emu3 eliminates diffusion and compositional architectures, using only next-token prediction for multimodal generation and perception.
将图像、文本和视频离散化为 token，从头训练单个 Transformer 处理混合序列。 Tokenizes images, text, and videos into discrete space, training a single transformer from scratch on mixed sequences.
在生成和理解基准上超越 SDXL 和 LLaVA-1.6，无需任务特定设计。 Outperforms SDXL and LLaVA-1.6 on generation and understanding benchmarks without task-specific designs.
通过预测下一个 token 生成高保真视频，实现自回归视频扩展。 Generates high-fidelity video by predicting next tokens, enabling autoregressive video extension.
将多模态模型简化为对 token 的统一关注，释放训练和推理的 Scaling 潜力。 Simplifies multimodal models to a single focus on tokens, unlocking scaling potential for both training and inference.
局限 · Limitations
依赖外部模型（GPT-4V、美学预测器）进行数据标注和筛选。 Relies on external models (GPT-4V, aesthetic predictors) for data captioning and filtering.
由于长上下文窗口（131k）和两阶段训练，计算成本高。 High computational cost due to large context length (131k) and two-stage training.
视频生成质量略低于顶级闭源模型如 Kling 和 Gen-3。 Video generation quality slightly below top closed-source models like Kling and Gen-3.
DPO 微调虽提升人类偏好，但可能导致某些自动指标下降。 DPO fine-tuning may degrade some automated metrics despite improving human preference.
模型规模固定，未探讨多模态数据下的缩放定律。 Model size fixed; ablation on scaling laws for multimodal data not explored.