We propose a simple pairwise Sigmoid loss for Language-Image Pre-training (SigLIP). Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. Combined with Locked-image Tuning, with only four TPUv4 chips, we train a SigLiT model that achieves 84.5% ImageNet zero-shot accuracy in two days. The disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient. We release our models at https://github.com/google-research/big_vision and hope our research motivates further explorations in improving the quality and efficiency of language-image pre-training.
核心贡献 · Key contributions
提出用于语言-图像预训练的成对 Sigmoid 损失，消除了全局归一化的需求。 Proposes a pairwise sigmoid loss for language-image pre-training, removing the need for global normalization.
在小批量大小下优于 Softmax 对比损失，并能高效扩展到大批量大小。 Outperforms softmax contrastive loss at small batch sizes and enables efficient scaling to large batch sizes.
结合锁定图像微调，仅用 4 块 TPUv4 芯片在 2 天内达到 84.5%的 ImageNet 零样本准确率。 Combined with locked-image tuning, achieves 84.5% ImageNet zero-shot accuracy with only 4 TPUv4 chips in 2 days.
表明批量大小的收益在 32k 处饱和，更大的批量大小收益递减。 Shows that batch size benefits saturate at 32k, with larger batch sizes yielding diminishing returns.
提供内存高效的分块实现，支持超过 100 万批量大小的训练。 Provides a memory-efficient chunked implementation that enables training with batch sizes over 1 million.
展示了对噪声数据的鲁棒性，并研究了正负样本对比例的影响。 Demonstrates robustness to noisy data and studies the impact of positive-negative pair ratio.
局限 · Limitations
在极大批量大小下，Sigmoid 和 Softmax 损失之间的性能差距缩小。 Performance gap between sigmoid and softmax loss diminishes at very large batch sizes.
Sigmoid 损失面临极端的正负样本不平衡，需要仔细的偏置初始化。 The sigmoid loss suffers from extreme positive-negative imbalance, requiring careful bias initialization.
在在批量大小下，即使对于中等模型也可能出现训练不稳定性。 Training instability may occur at large batch sizes, even for modest model sizes.
研究限于英语零样本任务；多语言结果显示超过 32k 批量大小无收益。 The study is limited to English zero-shot tasks; multilingual results show no benefit beyond 32k batch size.
锁定图像微调依赖于预训练的视觉塔，限制了在新架构上的适用性。 Locked-image tuning relies on a pre-trained vision tower, limiting applicability for new architectures.
论文章节 · Sections（共 16）
摘要Abstract
1 引言1 Introduction
相关工作2 Related Work
3 方法3 Method
3.1 语言-图像预训练的 Softmax 损失3.1 Softmax loss for language image pre-training