Recent self-supervised methods for image representation learning are based on maximizing the agreement between embedding vectors from different views of the same image. A trivial solution is obtained when the encoder outputs constant vectors. This collapse problem is often avoided through implicit biases in the learning architecture, that often lack a clear justification or interpretation. In this paper, we introduce VICReg (Variance-Invariance-Covariance Regularization), a method that explicitly avoids the collapse problem with a simple regularization term on the variance of the embeddings along each dimension individually. VICReg combines the variance term with a decorrelation mechanism based on redundancy reduction and covariance regularization, and achieves results on par with the state of the art on several downstream tasks. In addition, we show that incorporating our new variance term into other methods helps stabilize the training and leads to performance improvements.
核心贡献 · Key contributions
提出 VICReg，一种通过方差、不变性和协方差正则化显式防止坍塌的自监督方法。 Introduces VICReg, a self-supervised method with variance, invariance, and covariance regularization to explicitly prevent collapse.
方差项将嵌入方差维持在阈值以上，防止范数坍塌。 Variance term maintains embedding variance above a threshold, preventing norm collapse.
协方差项去相关嵌入维度，最大化信息含量。 Covariance term decorrelates embedding dimensions, maximizing information content.
VICReg 无需权重共享、批归一化或记忆库，支持多模态应用。 VICReg does not require weight sharing, batch normalization, or memory banks, enabling multi-modal applications.
在 ImageNet 线性评估和迁移任务上达到最先进水平。 Achieves state-of-the-art results on ImageNet linear evaluation and transfer tasks.
方差正则化加入其他方法时可稳定训练并提升性能。 Variance regularization stabilizes training and improves performance when added to other methods.
局限 · Limitations
在检测任务上略逊于 SwAV 和 BYOL 等顶尖方法。 Performance on detection tasks slightly behind top methods like SwAV and BYOL.
需要仔细调优超参数λ和μ以获得最佳性能。 Requires careful tuning of hyperparameters λ and μ for optimal performance.
协方差正则化与停止梯度结合时可能阻碍优化。 Covariance regularization may hinder optimization when combined with stop-gradient.
VICReg 的扩展器架构在预训练期间增加了计算开销。 VICReg's expander architecture adds computational overhead during pretraining.
除图像-文本检索外，多模态任务评估有限。 Limited evaluation on multi-modal tasks beyond image-text retrieval.
论文章节 · Sections（共 13）
摘要Abstract
1 引言1 Introduction
2 VICReg：直观理解2 VICReg: intuition
3 相关工作3 Related work
4 VICReg：详细描述4 VICReg: detailed description
4.1 方法4.1 Method
4.2 实现细节4.2 Implementation details
5 结果5 Results
5.1 ImageNet 评估5.1 Evaluation on ImageNet
5.2 迁移至其他下游任务5.2 Transfer to other downstream tasks
5.3 MS-COCO 多模态预训练5.3 Multi-modal pretraining on MS-COCO