Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. A successful approach to SSL is to learn embeddings which are invariant to distortions of the input sample. However, a recurring issue with this approach is the existence of trivial constant solutions. Most current methods avoid such solutions by careful implementation details. We propose an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible. This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors. The method is called Barlow Twins, owing to neuroscientist H. Barlow's redundancy-reduction principle applied to a pair of identical networks. Barlow Twins does not require large batches nor asymmetry between the network twins such as a predictor network, gradient stopping, or a moving average on the weight updates. Intriguingly it benefits from very high-dimensional output vectors. Barlow Twins outperforms previous methods on ImageNet for semi-supervised classification in the low-data regime, and is on par with current state of the art for ImageNet classification with a linear classifier head, and for transfer tasks of classification and object detection.
核心贡献 · Key contributions
提出 Barlow Twins，一种通过冗余减少避免平凡常数解的自监督学习方法。 Proposes Barlow Twins, a self-supervised learning method using redundancy reduction to avoid trivial constant solutions.
目标函数测量孪生网络输出之间的互相关矩阵，并使其接近单位矩阵。 Objective function measures cross-correlation matrix between twin network outputs and makes it close to identity matrix.
不需要大批量、预测网络、梯度停止或动量编码器，简化了实现。 Does not require large batches, predictor networks, stop-gradients, or momentum encoders, simplifying implementation.
受益于非常高维的输出向量，与饱和的对比方法不同。 Benefits from very high-dimensional output vectors, unlike contrastive methods that saturate.
在 ImageNet 半监督分类（1%标签）上优于先前方法（55% top-1 准确率）。 Outperforms prior methods on ImageNet semi-supervised classification with 1% labels (55% top-1 accuracy).
在 ImageNet 线性分类以及分类和物体检测的迁移任务上达到最先进水平。 Matches state-of-the-art on ImageNet linear classification and transfer tasks for classification and object detection.
局限 · Limitations
移除某些数据增强时性能下降，不像 BYOL 那样鲁棒。 Performance degrades when certain data augmentations are removed, unlike BYOL which is more robust.
需要仔细调整权衡参数λ以获得最佳性能。 Requires careful tuning of the trade-off parameter λ for optimal performance.