本文表明，掩码自编码器（MAE）是可扩展的计算机视觉自监督学习器。我们的 MAE 方法很简单：随机遮蔽输入图像的部分块，并重建缺失的像素。它基于两个核心设计。首先，我们开发了一种非对称的编码器-解码器架构，编码器仅对可见的块子集（无掩码标记）进行操作，而轻量级解码器则从潜在表示和掩码标记重建原始图像。其次，我们发现遮蔽输入图像的高比例（例如 75%）会产生一个非平凡且有意义的自监督任务。结合这两种设计，我们能够高效且有效地训练大型模型：我们加速了训练（3 倍或更多）并提高了准确性。我们的可扩展方法允许学习泛化良好的高容量模型：例如，一个普通的 ViT-Huge 模型在使用仅 ImageNet-1K 数据的方法中达到了最佳准确率（87.8%）。在下游任务中的迁移性能优于监督预训练，并显示出有希望的扩展行为。
This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3x or more) and improve accuracy. Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pre-training and shows promising scaling behavior.
核心贡献 · Key contributions
提出掩码自编码器（MAE）作为视觉中可扩展的自监督学习方法。 Proposes masked autoencoders (MAE) as scalable self-supervised learners for vision.
引入非对称编码器-解码器设计：编码器仅处理可见块，轻量解码器重建像素。 Introduces asymmetric encoder-decoder design: encoder only on visible patches, lightweight decoder reconstructs pixels.
表明高掩码比例（75%）创建了具有挑战性的自监督任务，减少了冗余。 Shows high masking ratio (75%) creates a challenging self-supervisory task that reduces redundancy.
在 ImageNet-1K 上使用原始 ViT-Huge 达到 87.8% top-1 准确率，优于先前方法。 Achieves 87.8% top-1 accuracy on ImageNet-1K with vanilla ViT-Huge, outperforming prior methods.
在检测和分割任务上展示了优于监督预训练的迁移学习增益。 Demonstrates transfer learning gains over supervised pre-training on detection and segmentation.
观察到有前景的缩放行为：更大的模型通过 MAE 预训练获得更好的性能。 Observes promising scaling behavior: larger models yield better performance with MAE pre-training.
局限 · Limitations
方法依赖随机块掩码，可能无法捕捉语义结构如物体边界。 Method relies on random patch masking, which may not capture semantic structure like object boundaries.
重建目标是像素，属于低级特征，可能限制高级语义学习。 Reconstruction target is pixels, which are low-level and may limit high-level semantic learning.
线性探测准确率低于对比方法，表明特征线性可分性较差。 Linear probing accuracy is lower than contrastive methods, indicating less linearly separable features.
需要较长的预训练周期（如 1600 轮）才能达到最佳性能。 Requires long pre-training schedules (e.g., 1600 epochs) for optimal performance.
模型可能反映数据集偏差并生成不存在的内容，引发社会担忧。 Model may reflect dataset biases and generate inexistent content, raising societal concerns.