本文探讨自监督学习是否为视觉 Transformer（ViT）带来区别于卷积网络（convnet）的新特性。除了自监督方法在此架构上表现尤佳外，我们观察到：首先，自监督 ViT 特征包含图像的显式语义分割信息，这在监督 ViT 或 convnet 中并不明显；其次，这些特征也是优秀的 k-NN 分类器，在小型 ViT 上 ImageNet top-1 准确率达 78.3%。研究还强调了动量编码器、多裁剪训练以及 ViT 中小 patch 的使用。我们将发现整合为简单自监督方法 DINO，可视为无标签的自蒸馏。通过 ViT-Base，DINO 在 ImageNet 线性评估中达到 80.1% top-1 准确率，展现了 DINO 与 ViT 的协同效应。
In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder, multi-crop training, and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.
核心贡献 · Key contributions
自监督 ViT 特征显式编码语义分割信息，这与有监督 ViT 或卷积网络不同。 Self-supervised ViT features explicitly encode semantic segmentation information, unlike supervised ViTs or convnets.
自监督 ViT 特征在 ImageNet 上使用简单的 k-NN 分类器达到 78.3%的 top-1 准确率。 Self-supervised ViT features achieve 78.3% top-1 accuracy on ImageNet with a simple k-NN classifier.
DINO 方法（一种无标签的自蒸馏形式）在 ViT-Base 的 ImageNet 线性评估中达到 80.1%的 top-1 准确率。 The DINO method, a form of self-distillation with no labels, achieves 80.1% top-1 on ImageNet linear evaluation with ViT-Base.
动量编码器、多裁剪训练和小 patch 对于高质量自监督 ViT 特征至关重要。 Momentum encoder, multi-crop training, and small patches are crucial for high-quality self-supervised ViT features.
自监督 ViT 特征在最近邻检索和复制检测任务中优于有监督特征。 Self-supervised ViT features outperform supervised ones in nearest neighbor retrieval and copy detection tasks.
DINO 与 ViT 结合在视频实例分割等密集任务上无需微调即达到竞争性能。 DINO with ViT achieves competitive performance on dense tasks like video instance segmentation without fine-tuning.
局限 · Limitations
研究聚焦于 ImageNet，未充分探索对其他数据集的泛化。 The study focuses on ImageNet; generalization to other datasets is not fully explored.
小 patch 提升性能但显著降低吞吐量并增加内存使用。 Small patches improve performance but significantly reduce throughput and increase memory usage.
DINO 需要仔细调整中心化和锐化以避免崩溃，这可能对超参数敏感。 DINO requires careful tuning of centering and sharpening to avoid collapse, which may be sensitive to hyperparameters.
该方法在 ViT 和 ResNet 上评估，未验证对其他架构的适用性。 The method is evaluated on ViT and ResNet; applicability to other architectures is not verified.
多裁剪带来的性能提升随视图数量增加而递减，表明收益递减。 Performance gains from multi-crop diminish with more views, suggesting diminishing returns.
论文章节 · Sections（共 15）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related work
3.1 基于知识蒸馏的自监督学习3.1 SSL with Knowledge Distillation
3.2 实现与评估协议3.2 Implementation and evaluation protocols
4 主要结果4 Main Results
4.1 在 ImageNet 上与自监督学习框架的比较4.1 Comparing with SSL frameworks on ImageNet
4.2 自监督学习训练的 ViT 特性4.2 Properties of ViT trained with SSL
5 DINO 的消融研究5 Ablation Study of DINO
5.1 不同组件的重要性5.1 Importance of the Different Components
5.2 教师网络选择的影响5.2 Impact of the choice of Teacher Network