Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models that learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skipgram model, where each word is represented as a bag of character $n$-grams. A vector representation is associated to each character $n$-gram; words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpora quickly and allows us to compute word representations for words that did not appear in the training data. We evaluate our word representations on nine different languages, both on word similarity and analogy tasks. By comparing to recently proposed morphological word representations, we show that our vectors achieve state-of-the-art performance on these tasks.
核心贡献 · Key contributions
提出一种基于字符 n-gram 的 skipgram 扩展，以捕捉子词信息。 Proposes a skipgram extension using character n-grams to capture subword information.
将每个词表示为其字符 n-gram 向量的和，实现参数共享。 Represents each word as a sum of its character n-gram vectors, enabling parameter sharing.
在九种语言的词相似性和类比任务上达到最先进性能。 Achieves state-of-the-art performance on word similarity and analogy tasks across nine languages.
通过组合字符 n-gram 向量处理词汇表外单词。 Handles out-of-vocabulary words by composing vectors from their character n-grams.
对小训练数据具有鲁棒性，在有限语料上优于基线模型。 Demonstrates robustness to small training data, outperforming baselines with limited corpora.
改善语言建模困惑度，尤其对形态丰富的语言效果显著。 Improves language modeling perplexity, especially for morphologically rich languages.
局限 · Limitations
在语义类比任务上，若 n-gram 长度选择不当，某些语言性能可能下降。 Performance on semantic analogy tasks may degrade for some languages with suboptimal n-gram lengths.
模型依赖固定的 n-gram 长度范围（3-6），可能并非所有语言的最优选择。 Model relies on a fixed range of n-gram lengths (3-6), which may not be optimal for all languages.
由于 n-gram 处理，训练速度比 skipgram 基线慢约 1.5 倍。 Training is approximately 1.5x slower than the skipgram baseline due to n-gram processing.
词相似性数据集可能包含词汇表外单词，基线模型需使用零向量。 Evaluation on word similarity datasets may include out-of-vocabulary words, requiring null vectors for baselines.
方法未使用形态分割，可能遗漏某些语言结构。 Method does not use morphological segmentation, potentially missing some linguistic structures.
论文章节 · Sections（共 22）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related work
3 模型3 Model
3.1 通用模型3.1 General model
3.2 子词模型3.2 Subword model
4.1 基线4.1 Baseline
4.2 优化4.2 Optimization
4.3 实现细节4.3 Implementation details
4.4 数据集4.4 Datasets
5 结果5 Results
5.1 人类相似性判断5.1 Human similarity judgement
5.2 词语类比任务5.2 Word analogy tasks
5.3 与形态表示的对比5.3 Comparison with morphological representations
5.4 训练数据规模的影响5.4 Effect of the size of the training data
5.5 n-gram 大小的影响5.5 Effect of the size of n𝑛n-grams
5.6 语言建模5.6 Language modeling
6.1 最近邻6.1 Nearest neighbors.
6.2 字符 n-gram 与语素6.2 Character n𝑛n-grams and morphemes