State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.
核心贡献 · Key contributions
提出 CLIP，一种对比式语言-图像预训练方法，从自然语言监督中大规模学习视觉表征。 Proposes CLIP, a contrastive language-image pre-training method that learns visual representations from natural language supervision at scale.
展示了对 30 多个数据集的零样本迁移能力，在 ImageNet 上无需任何训练样本即匹配 ResNet-50。 Demonstrates zero-shot transfer to over 30 datasets, matching ResNet-50 on ImageNet without any training examples.
表明零样本 CLIP 模型对自然分布偏移的有效鲁棒性显著高于有监督的 ImageNet 模型。 Shows that zero-shot CLIP models exhibit significantly higher effective robustness to natural distribution shifts than supervised ImageNet models.
引入了一个从互联网收集的 4 亿（图像，文本）对的新数据集用于预训练。 Introduces a new dataset of 400 million (image, text) pairs collected from the internet for pre-training.
发现零样本 CLIP 在相同特征空间上匹配 4 样本逻辑回归的性能，凸显自然语言监督的效率。 Finds that zero-shot CLIP matches the performance of 4-shot logistic regression on the same feature space, highlighting the efficiency of natural language supervision.
展示 CLIP 在预训练期间学习了广泛的任务，包括 OCR、地理定位和动作识别。 Demonstrates that CLIP learns a wide range of tasks during pre-training, including OCR, geo-localization, and action recognition.
局限 · Limitations
零样本 CLIP 在卫星图像分类和淋巴结肿瘤检测等专业任务上表现不佳。 Zero-shot CLIP underperforms on specialized tasks like satellite image classification and lymph node tumor detection.
研究聚焦于零样本和线性探测评估，而非端到端微调。 The study focuses on zero-shot and linear probe evaluation, not end-to-end fine-tuning.
预训练数据集可能包含来自互联网来源的偏见，影响模型公平性。 The pre-training dataset may contain biases from internet sources, affecting model fairness.
CLIP 的零样本性能受限于类别名称的多义性和缺乏上下文。 CLIP's zero-shot performance is limited by polysemy and lack of context in class names.
研究未探索零样本和少样本学习方法的结合。 The study does not explore the combination of zero-shot and few-shot learning methods.
论文章节 · Sections（共 19）
摘要Abstract
1 引言与动机工作1 Introduction and Motivating Work
2.1 自然语言监督2.1 Natural Language Supervision
2.2 创建足够大的数据集2.2 Creating a Sufficiently Large Dataset
2.3 选择高效的预训练方法2.3 Selecting an Efficient Pre-Training Method
2.4 选择与扩展模型2.4 Choosing and Scaling a Model
2.5 训练2.5 Training
3.1 零样本迁移3.1 Zero-Shot Transfer
3.2 表示学习3.2 Representation Learning
3.3 对自然分布变化的鲁棒性3.3 Robustness to Natural Distribution Shift