We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e, image patches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens). We first "tokenize" the original image into visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training objective is to recover the original visual tokens based on the corrupted image patches. After pre-training BEiT, we directly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder. Experimental results on image classification and semantic segmentation show that our model achieves competitive results with previous pre-training methods. For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K, significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains 86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%). The code and pretrained models are available at https://aka.ms/beit.
核心贡献 · Key contributions
提出掩码图像建模（MIM）以自监督方式预训练视觉 Transformer。 Proposes masked image modeling (MIM) to pretrain vision Transformers in a self-supervised manner.
从变分自编码器角度为 MIM 提供理论解释。 Provides a theoretical explanation of MIM from the perspective of variational autoencoder.
预训练 BEiT 并在图像分类和语义分割上开展广泛微调实验。 Pretrains BEiT and conducts extensive fine-tuning on image classification and semantic segmentation.
展示 BEiT 的自注意力机制无需人工标注即可区分语义区域和物体边界。 Shows self-attention in BEiT learns to distinguish semantic regions and object boundaries without human annotation.
大型 BEiT 在 ImageNet-1K 上达到 86.3%，超越在 ImageNet-22K 上监督预训练的 ViT-L（85.2%）。 Large-size BEiT achieves 86.3% on ImageNet-1K, surpassing ViT-L with supervised pre-training on ImageNet-22K (85.2%).
局限 · Limitations
预训练需要额外的图像分词器（dVAE）在额外数据上训练。 Pre-training requires a separate image tokenizer (dVAE) trained on additional data.
尽管使用离散词符，掩码图像建模仍可能关注低级细节。 Masked image modeling may still focus on low-level details despite using discrete tokens.
性能提升主要在 ImageNet 和 ADE20K 上展示；对其他任务的泛化性未充分探索。 Performance gains are demonstrated mainly on ImageNet and ADE20K; generalizability to other tasks is not fully explored.
该方法依赖块状掩码策略；其他掩码策略可能产生不同结果。 The method relies on blockwise masking strategy; other masking strategies may yield different results.
预训练计算成本高（16 块 GPU 上 500k 步需 5 天）。 Pre-training is computationally expensive (500k steps on 16 GPUs for 5 days).