The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.
核心贡献 · Key contributions
提出稀疏门控混合专家（MoE）层，实现超过 1000 倍的模型容量提升，仅带来轻微计算效率损失。 Introduces Sparsely-Gated Mixture-of-Experts (MoE) layer achieving >1000x capacity increase with minor efficiency loss.
在堆叠 LSTM 层之间卷积式应用 MoE，用于语言建模和机器翻译。 Applies MoE convolutionally between stacked LSTM layers for language modeling and translation.
展示了高达 1370 亿参数的 MoE 模型，以更低计算成本取得最先进结果。 Demonstrates up to 137 billion parameter MoE models with state-of-the-art results at lower computational cost.
解决了关键挑战：稀疏门控、负载均衡以及同步数据/模型并行的高效分布式训练。 Addresses key challenges: sparse gating, load balancing, and efficient distributed training with synchronous data/model parallelism.
显示专家在语法和语义上专门化，提升了模型可解释性。 Shows experts specialize in syntax and semantics, improving model interpretability.
在 10 亿词基准上困惑度降低 24%，在 1000 亿词语料上降低 39%。 Achieves 24% lower perplexity on 1B-word benchmark and 39% lower on 100B-word corpus compared to baselines.
局限 · Limitations
MoE 需要大批量数据和专门的分布式基础设施以维持效率。 MoE requires large batch sizes and specialized distributed infrastructure to maintain efficiency.
门控网络若无仔细正则化，可能收敛到不平衡的专家使用。 Gating network may converge to imbalanced expert usage without careful regularization.
方法主要在文本任务上验证，对较小数据集的其他领域适用性不确定。 Method primarily validated on text tasks; applicability to other domains with smaller datasets is uncertain.
层次化 MoE 增加复杂性，在极大专家数量下可能不如扁平 MoE 可扩展。 Hierarchical MoE adds complexity and may not scale as well as flat MoE for very large expert counts.
负载均衡损失引入了需要调优的额外超参数。 Load balancing losses introduce additional hyperparameters that require tuning.
论文章节 · Sections（共 14）
摘要Abstract
1.1 条件计算1.1 Conditional Computation
1.2 我们的方法：稀疏门控混合专家层1.2 Our Approach: The Sparsely-Gated Mixture-of-Experts Layer
1.3 混合专家模型的相关工作1.3 Related work on Mixtures of Experts
2 混合专家层的结构2 The Structure of the Mixture-of-Experts layer
2.1 门控网络2.1 Gating Network
3.1 批次缩小问题3.1 The Shrinking Batch Problem
3.2 网络带宽3.2 Network Bandwidth
4 平衡专家利用率4 Balancing Expert Utilization
5.1 10 亿词语言建模基准5.1 1 Billion Word Language Modeling Benchmark
5.2 1000 亿词谷歌新闻语料库5.2 100 Billion Word Google News Corpus
5.3 机器翻译（单一语言对）5.3 Machine Translation (Single Language Pair)