We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck. Given that natural language is highly context-dependent, this further implies that in practice Softmax with distributed word embeddings does not have enough capacity to model natural language. We propose a simple and effective method to address this issue, and improve the state-of-the-art perplexities on Penn Treebank and WikiText-2 to 47.69 and 40.68 respectively. The proposed method also excels on the large-scale 1B Word dataset, outperforming the baseline by over 5.6 points in perplexity.
核心贡献 · Key contributions
通过将语言建模形式化为矩阵分解问题，识别出 Softmax 瓶颈。 Identifies the Softmax bottleneck by formulating language modeling as a matrix factorization problem.
提出混合 Softmax（MoS）以打破基于 Softmax 模型的秩限制。 Proposes Mixture of Softmaxes (MoS) to break the rank limitation of Softmax-based models.
在 Penn Treebank（47.69）和 WikiText-2（40.68）上达到最先进的困惑度。 Achieves state-of-the-art perplexities on Penn Treebank (47.69) and WikiText-2 (40.68).
在大规模 1B Word 数据集上，困惑度比基线低 5.6 以上。 Outperforms baseline by over 5.6 perplexity points on the large-scale 1B Word dataset.
证明 MoS 学习高秩对数概率矩阵，而 Softmax 和 MoC 则不然。 Demonstrates that MoS learns high-rank log-probability matrices, unlike Softmax and MoC.
展示矩阵秩与语言建模性能之间的正相关关系。 Shows positive correlation between matrix rank and language modeling performance.
局限 · Limitations
MoS 因 K 倍更大的 Softmax 计算而引入计算开销。 MoS introduces computational overhead due to K-times larger Softmax computation.
理论分析假设上下文有限，而自然语言是无界的。 The theoretical analysis assumes a finite set of contexts, though natural language is unbounded.
实验表明，混合成分过多时 MoS 可能过拟合。 MoS may overfit when the number of mixture components is too large, as shown in experiments.
该方法仅在语言建模和对话任务上评估，未证明对其他领域的泛化性。 The method is evaluated only on language modeling and dialog tasks; generalizability to other domains is not proven.
秩估计依赖于数值近似，可能不精确。 The rank estimation relies on numerical approximations and may not be exact.
论文章节 · Sections（共 14）
摘要Abstract
1 引言1 Introduction
2 语言建模作为矩阵分解2 Language Modeling as Matrix Factorization
2.1 Softmax2.1 Softmax
2.2 假设：自然语言是高秩的2.2 Hypothesis: Natural Language is High-Rank
2.3 简单的修复？2.3 Easy Fixes?
2.4 Softmax 混合：一种高秩语言模型2.4 Mixture of Softmaxes: A High-Rank Language Model
2.5 上下文混合：一种低秩基线2.5 Mixture of Contexts: A Low-Rank Baseline