This paper presents SimMIM, a simple framework for masked image modeling. We simplify recently proposed related approaches without special designs such as block-wise masking and tokenization via discrete VAE or clustering. To study what let the masked image modeling task learn good representations, we systematically study the major components in our framework, and find that simple designs of each component have revealed very strong representation learning performance: 1) random masking of the input image with a moderately large masked patch size (e.g., 32) makes a strong pre-text task; 2) predicting raw pixels of RGB values by direct regression performs no worse than the patch classification approaches with complex designs; 3) the prediction head can be as light as a linear layer, with no worse performance than heavier ones. Using ViT-B, our approach achieves 83.8% top-1 fine-tuning accuracy on ImageNet-1K by pre-training also on this dataset, surpassing previous best approach by +0.6%. When applied on a larger model of about 650 million parameters, SwinV2-H, it achieves 87.1% top-1 accuracy on ImageNet-1K using only ImageNet-1K data. We also leverage this approach to facilitate the training of a 3B model (SwinV2-G), that by $40\times$ less data than that in previous practice, we achieve the state-of-the-art on four representative vision benchmarks. The code and models will be publicly available at https://github.com/microsoft/SimMIM.
核心贡献 · Key contributions
提出 SimMIM，一个简单的掩码图像建模框架，无需块状掩码或离散 VAE 分词等特殊设计。 Proposes SimMIM, a simple framework for masked image modeling without special designs like block-wise masking or discrete VAE tokenization.
表明使用大块尺寸（如 32）的随机掩码和通过ℓ1 损失的原始像素回归可产生强表征。 Shows random masking with large patch size (e.g., 32) and raw pixel regression via ℓ1 loss yields strong representations.
证明线性预测头足够，在 ImageNet-1K 上使用 ViT-B 达到 83.8% top-1 准确率。 Demonstrates a linear prediction head suffices, achieving 83.8% top-1 accuracy on ImageNet-1K with ViT-B.
扩展到 3B 参数的 SwinV2-G，在四个视觉基准上以 40 倍更少数据达到最先进水平。 Scales to 3B-parameter SwinV2-G, achieving state-of-the-art on four vision benchmarks with 40× less data.
系统研究掩码策略、预测目标和头部设计，揭示简单设计最优。 Systematically studies masking strategy, prediction target, and head design, revealing simple designs are optimal.
在 https://github.com/microsoft/SimMIM 公开提供代码和模型。 Provides code and models publicly at https://github.com/microsoft/SimMIM.
局限 · Limitations
主要评估在 ImageNet-1K 上；未广泛验证对其他领域的泛化性。 Evaluation primarily on ImageNet-1K; generalization to other domains not extensively validated.
需要大掩码块尺寸（如 32）和高掩码率（如 60%）以获得最佳性能。 Requires large masked patch size (e.g., 32) and high masking ratio (e.g., 60%) for best performance.
仅在 ImageNet-1K 上预训练；在其他数据集上的性能可能不同。 Pre-training on ImageNet-1K only; performance on other datasets may vary.
更重的预测头或更高目标分辨率不会提升微调性能。 Heavier prediction heads or higher target resolutions do not improve fine-tuning performance.
对简单设计为何优于复杂设计的分析有限；依赖经验观察。 Limited analysis on why simple designs outperform complex ones; relies on empirical observations.
论文章节 · Sections（共 20）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3.1 掩码图像建模框架3.1 A Masked Image Modeling Framework
3.2 掩码策略3.2 Masking Strategy
3.3 预测头3.3 Prediction Head
3.4 预测目标3.4 Prediction Targets
3.5 评估协议3.5 Evaluation protocols
4.1 消融研究4.1 Ablation Study
4.2 与 ViT-B 上先前方法的比较4.2 Comparison to Previous Approaches on ViT-B
4.3 使用 Swin Transformer 的缩放实验4.3 Scaling Experiments with Swin Transformer
4.4 可视化4.4 Visualization
5 结论5 Conclusion
致谢Acknowledgement
附录 A 详细架构Appendix A Detailed Architectures
附录 B 学习率调度器的影响Appendix B The Effect of Learning Rate Schedulers
附录 C 下游任务结果Appendix C Results on Downstream Tasks
C.1 详细设置C.1 Detailed Settings
C.2 消融研究C.2 Ablation Studies
附录 E 使用 ConvNets 的 SimMIMAppendix E SimMIM with ConvNets