通过对比预训练学习文本和代码嵌入

Text and Code Embeddings by Contrastive Pre-Training

亚历克·拉德福德 Alec Radford · OpenAI · 2022-01-24 · arXiv:2201.10005 ↗ · 被引 601

摘要 · Abstract

文本嵌入在语义搜索和文本相似度计算等许多应用中都是有用的特征。以往的工作通常针对不同用例定制训练模型，在数据集选择、训练目标和模型架构上各不相同。在这项工作中，我们表明，在大规模无监督数据上进行对比预训练可以产生高质量的文本和代码向量表示。相同的无监督文本嵌入在线性探测分类中取得了新的最优结果，同时展现出令人印象深刻的语义搜索能力，有时甚至与微调模型竞争。在线性探测分类准确率上，平均 7 个任务，我们的最佳无监督模型相对于之前最佳的无监督和有监督文本嵌入模型分别实现了 4%和 1.8%的相对提升。相同的文本嵌入在大规模语义搜索评估中，在 MSMARCO、Natural Questions 和 TriviaQA 基准测试上，相对于之前最佳的无监督方法分别实现了 23.4%、14.7%和 10.6%的相对提升。与文本嵌入类似，我们在（文本，代码）对上训练代码嵌入模型，在代码搜索上相对于之前的最佳工作获得了 20.8%的相对提升。

Text embeddings are useful features in many applications such as semantic search and computing text similarity. Previous work typically trains models customized for different use cases, varying in dataset choice, training objective and model architecture. In this work, we show that contrastive pre-training on unsupervised data at scale leads to high quality vector representations of text and code. The same unsupervised text embeddings that achieve new state-of-the-art results in linear-probe classification also display impressive semantic search capabilities and sometimes even perform competitively with fine-tuned models. On linear-probe classification accuracy averaging over 7 tasks, our best unsupervised model achieves a relative improvement of 4% and 1.8% over previous best unsupervised and supervised text embedding models respectively. The same text embeddings when evaluated on large-scale semantic search attains a relative improvement of 23.4%, 14.7%, and 10.6% over previous best unsupervised methods on MSMARCO, Natural Questions and TriviaQA benchmarks, respectively. Similarly to text embeddings, we train code embedding models on (text, code) pairs, obtaining a 20.8% relative improvement over prior best work on code search.

核心贡献 · Key contributions

表明在无监督数据上进行大规模对比预训练可产生高质量的文本和代码嵌入。
Shows contrastive pre-training on unsupervised data at scale yields high-quality text and code embeddings.
在线性探测分类上取得最优结果，相对先前无监督方法提升 4%。
Achieves state-of-the-art linear-probe classification with 4% relative improvement over prior unsupervised methods.
在 MSMARCO、Natural Questions 和 TriviaQA 语义搜索上分别相对提升 23.4%、14.7%和 10.6%。
Attains 23.4%, 14.7%, and 10.6% relative improvement on MSMARCO, Natural Questions, and TriviaQA semantic search.
代码嵌入在 CodeSearchNet 代码搜索基准上相对提升 20.8%。
Code embeddings achieve 20.8% relative improvement on CodeSearchNet code search benchmark.
在 SST-2 上展示零样本和 k-NN 分类与监督方法竞争的性能。
Demonstrates zero-shot and k-NN classification competitive with supervised methods on SST-2.
微调模型在 NLI 任务上超越先前迁移方法 2.2%。
Fine-tuned models outperform previous transfer methods by 2.2% on NLI tasks.

局限 · Limitations

在句子相似性任务上表现不如先前最优方法。
Underperforms on sentence similarity tasks compared to prior state-of-the-art.
需要大批量大小和大量计算资源进行训练。
Requires large batch sizes and substantial computational resources for training.
代码嵌入在参数从 3 亿增至 12 亿时未见性能提升。
Code embeddings show no performance gain with increased model size from 300M to 1.2B parameters.
训练数据中的潜在偏见可能在下游应用中造成表征伤害。
Potential biases in training data may cause representational harms in downstream applications.
句子相似性和搜索任务可能目标冲突，限制联合优化。
Sentence similarity and search tasks may have conflicting objectives, limiting joint optimization.

论文章节 · Sections（共 13）

摘要 Abstract
1 引言 1 Introduction
2 方法 2 Approach
2.1 模型 2.1 Model
2.2 训练目标 2.2 Training Objective
3 结果 3 Results
3.1 文本嵌入 3.1 Text Embedding
3.2 文本搜索 3.2 Text Search
3.3 代码搜索 3.3 Code Search
3.4 分析 3.4 Analysis
4 相关工作 4 Related Work
5 广泛影响 5 Broader Impacts
6 结论 6 Conclusion

阅读逐段中英对照全文 →