The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.
核心贡献 · Key contributions
提出负采样作为层级 Softmax 的简单替代方案，实现高效训练。 Introduced negative sampling as a simple alternative to hierarchical softmax for efficient training.
证明对高频词进行下采样可加速训练并提升罕见词的向量质量。 Demonstrated that subsampling frequent words speeds up training and improves vector quality for rare words.
通过数据驱动的短语检测将短语视为单个词元，扩展 Skip-gram 以学习短语向量。 Extended Skip-gram to learn phrase vectors by treating phrases as single tokens via data-driven phrase detection.
展示词向量具有线性结构，可通过向量算术进行类比推理。 Showed that word vectors exhibit linear structure enabling analogical reasoning via vector arithmetic.
证明词向量的加法组合性，例如 vec('Russia') + vec('river') ≈ vec('Volga River')。 Demonstrated additive compositionality of word vectors, e.g., vec('Russia') + vec('river') ≈ vec('Volga River').
在数十亿词上训练高质量向量，在类比任务上显著优于先前模型。 Trained high-quality vectors on billions of words, significantly outperforming prior models on analogy tasks.
局限 · Limitations
词表示忽略词序，限制了对句法结构的捕捉。 Word representations are indifferent to word order, limiting capture of syntactic structure.
无法将习语短语表示为单个词义的组合。 Inability to represent idiomatic phrases as compositions of individual word meanings.
负采样不最大化 Softmax 的对数概率，仅近似之。 Negative sampling does not maximize log probability of softmax, only approximates it.
短语检测阈值和下采样率是启发式的且依赖任务。 Phrase detection threshold and subsampling rate are heuristic and task-dependent.
模型性能严重依赖大规模训练数据和细致的超参数调优。 Model performance heavily depends on large training data and careful hyperparameter tuning.
论文章节 · Sections（共 12）
摘要Abstract
1 引言1 Introduction
2 Skip-gram 模型2 The Skip-gram Model
2.1 层次 Softmax2.1 Hierarchical Softmax
2.2 负采样2.2 Negative Sampling
2.3 高频词下采样2.3 Subsampling of Frequent Words
3 实验结果3 Empirical Results
4 短语学习4 Learning Phrases
4.1 短语 Skip-gram 结果4.1 Phrase Skip-Gram Results
5 加法组合性5 Additive Compositionality
6 与已发表词向量的比较6 Comparison to Published Word Representations