Mixture of Experts (MoE) models have emerged as a promising paradigm for scaling language models efficiently by activating only a subset of parameters for each input token. In this report, we present dots.llm1, a large-scale MoE model that activates 14B parameters out of a total of 142B parameters, delivering performance on par with state-of-the-art models while reducing training and inference costs. Leveraging our meticulously crafted and efficient data processing pipeline, dots.llm1 achieves performance comparable to Qwen2.5-72B after pretraining on 11.2T high-quality tokens and post-training to fully unlock its capabilities. Notably, no synthetic data is used during pretraining. To foster further research, we open-source intermediate training checkpoints at every one trillion tokens, providing valuable insights into the learning dynamics of large language models.
核心贡献 · Key contributions
提出可扩展的三阶段数据处理框架，用于生成高质量、多样化的预训练数据。 Proposes a scalable three-stage data processing framework for generating high-quality, diverse pretraining data.
推出 dots.llm1，一个 142B MoE 模型，激活 14B 参数，性能媲美 Qwen2.5-72B 且成本更低。 Introduces dots.llm1, a 142B MoE model activating 14B parameters, achieving performance comparable to Qwen2.5-72B with reduced costs.
开发了基于 1F1B 流水线调度的创新 MoE 全到全通信与计算重叠方案。 Develops an innovative MoE all-to-all communication and computation overlap recipe based on 1F1B pipeline scheduling.
每万亿 token 开源中间训练检查点，提供学习动态洞察。 Open-sources intermediate training checkpoints at every one trillion tokens to provide insights into learning dynamics.
在单节点八 GPU 上实现高效推理，展示了成本效益。 Achieves efficient inference on a single node with eight GPUs, demonstrating cost-effectiveness.
在中文和英文任务上表现稳健，在中文语言和数学推理方面具有优势。 Demonstrates robust performance across Chinese and English tasks, with strengths in Chinese language and mathematical reasoning.
局限 · Limitations
编码性能不如其他领域突出，与前沿模型相比仍有提升空间。 Coding performance is not as exceptional as other areas, with room for improvement compared to cutting-edge models.
模型依赖高质量数据处理流程；低质量数据可能导致性能下降。 Model relies on high-quality data processing pipeline; performance may degrade with lower-quality data.
评估限于基准测试；实际部署性能可能有所不同。 Evaluation limited to benchmarks; real-world deployment performance may vary.
上下文长度仅扩展到 32K；更长的上下文可能无法有效处理。 Context length extension only up to 32K; longer contexts may not be handled effectively.
预训练未使用合成数据，可能限制某些领域的数据多样性。 No synthetic data used in pretraining, which may limit data diversity for certain domains.
论文章节 · Sections（共 16）
摘要Abstract
1 引言1 Introduction
2 架构2 Architecture
3 基础设施3 Infrastructures
3.1 基于交错 1F1B 的通信与计算重叠3.1 Interleaved 1F1B based Communication and Computation Overlap
3.2 分组 GEMM 的高效实现3.2 Efficient Implementation of Grouped GEMM