The success of language Transformers is primarily attributed to the pretext task of masked language modeling (MLM), where texts are first tokenized into semantically meaningful pieces. In this work, we study masked image modeling (MIM) and indicate the advantages and challenges of using a semantically meaningful visual tokenizer. We present a self-supervised framework iBOT that can perform masked prediction with an online tokenizer. Specifically, we perform self-distillation on masked patch tokens and take the teacher network as the online tokenizer, along with self-distillation on the class token to acquire visual semantics. The online tokenizer is jointly learnable with the MIM objective and dispenses with a multi-stage training pipeline where the tokenizer needs to be pre-trained beforehand. We show the prominence of iBOT by achieving an 82.3% linear probing accuracy and an 87.8% fine-tuning accuracy evaluated on ImageNet-1K. Beyond the state-of-the-art image classification results, we underline emerging local semantic patterns, which helps the models to obtain strong robustness against common corruptions and achieve leading results on dense downstream tasks, eg., object detection, instance segmentation, and semantic segmentation.
核心贡献 · Key contributions
提出 iBOT，一种使用在线分词器的自监督掩码图像建模框架。 Proposes iBOT, a self-supervised framework for masked image modeling with an online tokenizer.
在 ImageNet-1K 上达到 82.3%线性探测和 87.8%微调准确率，均为最优。 Achieves state-of-the-art linear probing (82.3%) and fine-tuning (87.8%) accuracy on ImageNet-1K.
通过自蒸馏联合学习分词器和目标模型，消除了多阶段训练。 Eliminates multi-stage training by jointly learning the tokenizer and target model via self-distillation.
发现补丁令牌中涌现的局部语义，增强了鲁棒性和密集任务性能。 Discovers emerging part-level semantics in patch tokens, enhancing robustness and dense task performance.
在目标检测、实例分割和语义分割上超越先前方法。 Outperforms prior methods on object detection, instance segmentation, and semantic segmentation.
展示了向更大模型和数据集的扩展性，持续优于 DINO。 Demonstrates scalability to larger models and datasets, with consistent gains over DINO.
局限 · Limitations
训练更大模型需要更多数据，ViT-L/16 在 1K 数据上不如 BEiT。 Requires more data to train larger models, as ViT-L/16 underperforms BEiT on 1K data.
性能依赖于共享投影头设计；分离头导致结果较差。 Performance depends on the shared projection head design; separate heads yield worse results.
对 ImageNet 以外数据集的评估有限；迁移学习结果可能不同。 Limited evaluation on datasets beyond ImageNet; transfer learning results may vary.
在线分词器在某些领域可能不如离线分词器有效捕捉语义。 Online tokenizer may not capture semantics as effectively as offline tokenizers in some domains.
动量更新的自蒸馏计算成本对超大模型可能较高。 Computational cost of self-distillation with momentum update may be high for very large models.
论文章节 · Sections（共 14）
摘要Abstract
1 引言1 Introduction
2.1 掩码图像建模作为知识蒸馏2.1 Masked Image Modeling as Knowledge Distillation
2.2 自蒸馏2.2 Self-Distillation
3 iBOT3 iBOT
3.1 框架3.1 Framework
3.2 实现3.2 Implementation
4 实验4 Experiment
4.1 ImageNet-1K 分类4.1 Classification on ImageNet-1K
4.2 下游任务4.2 Downstream Tasks
4.3 MIM 训练的 ViT 特性4.3 Properties of ViT trained with MIM