In this technical report, we tackle the challenges of training large-scale Mixture of Experts (MoE) models, focusing on overcoming cost inefficiency and resource limitations prevalent in such systems. To address these issues, we present two differently sized MoE large language models (LLMs), namely Ling-Lite and Ling-Plus (referred to as "Bailing" in Chinese, spelled Bǎilíng in Pinyin). Ling-Lite contains 16.8 billion parameters with 2.75 billion activated parameters, while Ling-Plus boasts 290 billion parameters with 28.8 billion activated parameters. Both models exhibit comparable performance to leading industry benchmarks. This report offers actionable insights to improve the efficiency and accessibility of AI development in resource-constrained settings, promoting more scalable and sustainable technologies. Specifically, to reduce training costs for large-scale MoE models, we propose innovative methods for (1) optimization of model architecture and training processes, (2) refinement of training anomaly handling, and (3) enhancement of model evaluation efficiency. Additionally, leveraging high-quality data generated from knowledge graphs, our models demonstrate superior capabilities in tool use compared to other models. Ultimately, our experimental findings demonstrate that a 300B MoE LLM can be effectively trained on lower-performance devices while achieving comparable performance to models of a similar scale, including dense and MoE models. Compared to high-performance devices, utilizing a lower-specification hardware system during the pre-training phase demonstrates significant cost savings, reducing computing costs by approximately 20%. The models can be accessed at https://huggingface.co/inclusionAI.
核心贡献 · Key contributions
提出创新方法降低大规模 MoE 模型训练成本，包括架构优化、训练流程、异常处理和评估效率。 Proposes innovative methods to reduce training costs for large-scale MoE models, including optimization of architecture, training processes, anomaly handling, and evaluation efficiency.
证明 300B MoE 大语言模型可在低性能设备上有效训练，性能与同规模模型相当，成本节省约 20%。 Demonstrates that a 300B MoE LLM can be effectively trained on lower-performance devices, achieving comparable performance to similar-scale models with ~20% cost savings.
开发 XPUTimer 轻量级调试工具，在异构训练中实现 90%内存减少用于性能分析。 Develops XPUTimer, a lightweight debugging tool achieving 90% memory reduction for performance analysis in heterogeneous training.
实现 EDiT 异步训练策略，在异构环境中训练时间最多减少 66.1%。 Implements EDiT asynchronous training strategy, reducing training time by up to 66.1% in heterogeneous environments.
通过基于知识图谱的高质量数据合成和自适应工具学习增强工具使用能力。 Enhances tool use capability via knowledge graph-based high-quality data synthesis and adaptive tool learning.
局限 · Limitations
尽管有缓解策略，训练稳定性问题如损失尖峰和专家负载不平衡仍然存在。 Training stability issues such as loss spikes and expert load imbalance persist despite mitigation strategies.
跨平台对齐仍具挑战，因异构硬件上精度误差累积。 Cross-platform alignment remains challenging due to accumulated precision errors across heterogeneous hardware.
研究聚焦 MoE 模型，结论可能不适用于稠密架构或其他模型家族。 The study focuses on MoE models; findings may not generalize to dense architectures or other model families.
约 20%的成本节省基于特定硬件配置，不同设置下可能有所变化。 Cost savings of ~20% are based on specific hardware configurations and may vary in different settings.
模型在有限基准上评估，更广泛任务上的性能有待验证。 The models are evaluated on limited benchmarks; performance on broader tasks remains to be validated.
论文章节 · Sections（共 27）
摘要Abstract
1.1 背景与动机1.1 Background and Motivation
1.2 模型训练的计算环境1.2 Computing Environment for Model Training
1.3 模型训练的优化1.3 Optimization for Model Training
1.4 挑战与经验教训1.4 Challenges and Lessons Learned
2 基础设施、扩展与效率2 Infrastructure, Scaling, and Efficiency
2.1 轻量级分析器2.1 Lightweight Profiler
2.2 高性能训练策略2.2 High-Performance Training Strategy
2.3 高效高可靠的跨集群数据同步2.3 Efficient and Highly-Reliable Cross-Cluster Data Synchronization