In this report, we introduce ERNIE 5.0, a natively autoregressive foundation model desinged for unified multimodal understanding and generation across text, image, video, and audio. All modalities are trained from scratch under a unified next-group-of-tokens prediction objective, based on an ultra-sparse mixture-of-experts (MoE) architecture with modality-agnostic expert routing. To address practical challenges in large-scale deployment under diverse resource constraints, ERNIE 5.0 adopts a novel elastic training paradigm. Within a single pre-training run, the model learns a family of sub-models with varying depths, expert capacities, and routing sparsity, enabling flexible trade-offs among performance, model size, and inference latency in memory- or time-constrained scenarios. Moreover, we systematically address the challenges of scaling reinforcement learning to unified foundation models, thereby guaranteeing efficient and stable post-training under ultra-sparse MoE architectures and diverse multimodal settings. Extensive experiments demonstrate that ERNIE 5.0 achieves strong and balanced performance across multiple modalities. To the best of our knowledge, among publicly disclosed models, ERNIE 5.0 represents the first production-scale realization of a trillion-parameter unified autoregressive model that supports both multimodal understanding and generation. To facilitate further research, we present detailed visualizations of modality-agnostic expert routing in the unified model, alongside comprehensive empirical analysis of elastic training, aiming to offer profound insights to the community.
核心贡献 · Key contributions
提出 ERNIE 5.0，一个支持文本、图像、视频和音频多模态理解与生成的万亿参数统一自回归模型。 Proposes ERNIE 5.0, a trillion-parameter unified autoregressive model supporting multimodal understanding and generation across text, image, video, and audio.
引入超稀疏混合专家（MoE）架构与模态无关的专家路由，实现高效的跨模态参数共享。 Introduces an ultra-sparse mixture-of-experts (MoE) architecture with modality-agnostic expert routing for efficient cross-modal parameter sharing.
开发弹性训练范式，在单次预训练中生成具有不同深度、宽度和稀疏度的子模型族。 Develops an elastic training paradigm that produces a family of sub-models with varying depth, width, and sparsity within a single pre-training run.
解决将强化学习扩展到统一基础模型的问题，采用无偏重放缓冲区和自适应提示强化学习等技术。 Addresses scaling reinforcement learning to unified foundation models with techniques like unbiased replay buffer and adaptive hint-based RL.
在多种模态上实现强劲且均衡的性能，匹配或超越专用基线模型。 Achieves strong and balanced performance across multiple modalities, matching or surpassing specialized baselines.
提供模态无关专家路由和弹性训练的详细可视化与实证分析。 Provides detailed visualizations and empirical analysis of modality-agnostic expert routing and elastic training.
局限 · Limitations
评估仅限于公开披露的模型，与专有模型的比较可能不完整。 Evaluation is limited to publicly disclosed models; comparisons with proprietary models may be incomplete.
弹性训练可能在子模型性能与完整模型质量之间引入权衡。 Elastic training may introduce trade-offs between sub-model performance and full-model quality.
统一自回归框架在处理极长多模态序列时可能面临挑战。 The unified autoregressive framework may face challenges in handling extremely long multimodal sequences.
强化学习训练稳定性依赖于复杂的算法和工程优化，可能无法泛化到所有场景。 RL training stability relies on complex algorithmic and engineering optimizations, which may not generalize to all settings.
模型在低资源语言或小众模态上的性能未得到广泛评估。 The model's performance on low-resource languages or niche modalities is not extensively evaluated.
论文章节 · Sections（共 25）
1 引言1 Introduction
2 架构2 Architecture
2.1 统一自回归骨干与超稀疏混合专家2.1 Unified Autoregressive Backbone with Ultra-Sparse Mixture-of-Experts
2.2 视觉建模2.2 Visual Modeling
2.3 音频建模2.3 Audio Modeling
3 预训练3 Pre-Training
3.1 预训练数据3.1 Pre-Training Data
3.2 训练方案3.2 Training Recipe
3.3 弹性训练的通用框架3.3 Once-For-All with Elastic Training
4 后训练4 Post-Training
4.1 利用无偏重放缓冲区提升推理效率4.1 Enhancing Rollout Efficiency with Unbiased Replay Buffer
4.2 缓解熵坍缩以稳定训练4.2 Stabilizing Training with Mitigated Entropy Collapse
4.3 基于提示学习提升样本效率4.3 Boosting Sample Efficiency with Hint-based Learning
5 基础设施5 Infrastructures
5.1 大规模训练的混合并行5.1 Hybrid Parallelism for Training at Scale
5.2 多模态训练的分离架构5.2 Disaggregation Architecture for Multimodal Training
5.3 灵活多模态注意力的 FlashMask5.3 FlashMask for Flexible Multimodal Attention
5.4 可扩展且分离的强化学习基础设施5.4 Scalable and Disaggregated RL Infrastructure