In this paper, we introduce Janus, an autoregressive framework that unifies multimodal understanding and generation. Prior research often relies on a single visual encoder for both tasks, such as Chameleon. However, due to the differing levels of information granularity required by multimodal understanding and generation, this approach can lead to suboptimal performance, particularly in multimodal understanding. To address this issue, we decouple visual encoding into separate pathways, while still leveraging a single, unified transformer architecture for processing. The decoupling not only alleviates the conflict between the visual encoder's roles in understanding and generation, but also enhances the framework's flexibility. For instance, both the multimodal understanding and generation components can independently select their most suitable encoding methods. Experiments show that Janus surpasses previous unified model and matches or exceeds the performance of task-specific models. The simplicity, high flexibility, and effectiveness of Janus make it a strong candidate for next-generation unified multimodal models.
核心贡献 · Key contributions
提出 Janus，一种自回归框架，通过解耦视觉编码实现多模态理解与生成的统一。 Proposes Janus, an autoregressive framework that decouples visual encoding for multimodal understanding and generation.
将视觉编码解耦为独立路径，以解决理解与生成任务之间的冲突。 Decouples visual encoding into separate pathways to resolve conflicts between understanding and generation tasks.
在多模态理解和生成基准上取得最先进性能，超越统一模型和专用模型。 Achieves state-of-the-art performance on multimodal understanding and generation benchmarks, surpassing unified and task-specific models.
展示高灵活性和可扩展性，允许为每个任务独立选择最优编码方法。 Demonstrates high flexibility and extensibility, allowing independent selection of optimal encoding methods for each task.
通过大量消融研究验证了解耦视觉编码的重要性。 Provides extensive ablation studies validating the importance of decoupling visual encoding.
展示了集成点云、脑电图或音频等额外模态的潜力。 Shows potential for integrating additional modalities like point clouds, EEG, or audio.
局限 · Limitations
模型规模限于 1.3B 参数；扩展到更大模型可能产生不同结果。 Model size limited to 1.3B parameters; scaling to larger models may yield different results.
评估主要针对静态图像；未探索视频理解与生成。 Evaluation primarily on static images; video understanding and generation not explored.
训练数据限于英文文本-图像对；多语言生成依赖 LLM 的固有能力。 Training data limited to English text-to-image pairs; multilingual generation relies on LLM's inherent abilities.
视觉生成质量在细粒度细节上可能落后于专用扩散模型。 Visual generation quality may lag behind specialized diffusion models in fine-grained details.
消融研究使用了主实验未用的语义分词器，可能影响可比性。 Ablation study uses a semantic tokenizer not used in main experiments, potentially affecting comparability.
论文章节 · Sections（共 23）
摘要Abstract
1 引言1 Introduction
2.1 视觉生成2.1 Visual Generation
2.2 多模态理解2.2 Multimodal Understanding
2.3 统一多模态理解与生成2.3 Unified Multimodal Understanding and Generation
3 Janus：一个简单、统一且灵活的多模态框架3 Janus: A Simple, Unified and Flexible Multimodal Framework
3.1 架构3.1 Architecture
3.2 训练流程3.2 Training Procedure
3.3 训练目标3.3 Training Objective
3.4 推理3.4 Inference
3.5 可能的扩展3.5 Possible Extensions
4 实验4 Experiments
4.1 实现细节4.1 Implementation Details
4.2 数据设置4.2 Data Setup
4.3 评估设置4.3 Evaluation Setup
4.4 与最新技术的比较4.4 Comparison with State-of-the-arts
4.5 消融研究4.5 Ablation Studies
4.6 定性结果4.6 Qualitative Results
5 结论5 Conclusion
A.1 语义分词器架构A.1 Architecture of Semantic Tokenizer
A.2 训练A.2 Training
A.3 与 LLM 集成A.3 Integrating with LLM
附录 B 额外定性结果Appendix B Additional Qualitative Results