We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities.
核心贡献 · Key contributions
提出 CBOW 和 Skip-gram 模型，可从大规模数据高效学习高质量词向量。 Proposes CBOW and Skip-gram models for efficient learning of high-quality word vectors from large data.
在句法和语义词相似性任务上以低计算成本达到最先进精度。 Achieves state-of-the-art accuracy on syntactic and semantic word similarity tasks with low computational cost.
证明词向量捕捉线性规律，支持代数运算如 vector("King") - vector("Man") + vector("Woman") ≈ vector("Queen")。 Demonstrates that word vectors capture linear regularities, enabling algebraic operations like vector("King") - vector("Man") + vector("Woman") ≈ vector("Queen").
引入包含 8869 个语义和 10675 个句法问题的综合测试集，用于评估词向量。 Introduces a comprehensive test set with 8869 semantic and 10675 syntactic questions for evaluating word vectors.
表明同时增加向量维度和训练数据量会带来递减收益，需要平衡缩放。 Shows that increasing both vector dimensionality and training data size yields diminishing returns, requiring balanced scaling.
通过结合 Skip-gram 与 RNNLM，在微软句子补全挑战中达到新最先进水平（58.9%准确率）。 Achieves new state-of-the-art on Microsoft Sentence Completion Challenge (58.9% accuracy) by combining Skip-gram with RNNLM.
局限 · Limitations
模型将词视为原子单元，忽略形态学；同义词在评估中被计为错误。 Models treat words as atomic units, ignoring morphology; synonyms are counted as mistakes in evaluation.
性能随向量维度或数据量增加而饱和，需要联合缩放才能改进。 Performance saturates with increasing vector dimensions or data size, requiring joint scaling for improvement.
Skip-gram 在句法任务上表现不如 CBOW，限制了其通用性。 Skip-gram model underperforms on syntactic tasks compared to CBOW, limiting its general applicability.
在有限的领域数据（如语音识别）上训练可能无法从简单缩放中受益，需要更高级技术。 Training on limited in-domain data (e.g., speech recognition) may not benefit from simple scaling, requiring advanced techniques.
评估指标（余弦距离）可能无法捕捉所有语义细微差别，100%准确率可能无法达到。 Evaluation metric (cosine distance) may not capture all semantic nuances, and 100% accuracy is likely unattainable.
论文章节 · Sections（共 20）
摘要Abstract
1 引言1 Introduction
1.1 本文目标1.1 Goals of the Paper
1.2 先前工作1.2 Previous Work
2 模型架构2 Model Architectures
2.1 前馈神经网络语言模型（NNLM）2.1 Feedforward Neural Net Language Model (NNLM)
2.2 循环神经网络语言模型（RNNLM）2.2 Recurrent Neural Net Language Model (RNNLM)
2.3 神经网络的并行训练2.3 Parallel Training of Neural Networks
3 新的对数线性模型3 New Log-linear Models
3.1 连续词袋模型3.1 Continuous Bag-of-Words Model
3.2 连续 Skip-gram 模型3.2 Continuous Skip-gram Model
4 结果4 Results
4.1 任务描述4.1 Task Description
4.2 准确率最大化4.2 Maximization of Accuracy
4.3 模型架构比较4.3 Comparison of Model Architectures
4.4 模型的大规模并行训练4.4 Large Scale Parallel Training of Models
4.5 微软研究院句子补全挑战4.5 Microsoft Research Sentence Completion Challenge