This paper demonstrates an approach for learning highly semantic image representations without relying on hand-crafted data-augmentations. We introduce the Image-based Joint-Embedding Predictive Architecture (I-JEPA), a non-generative approach for self-supervised learning from images. The idea behind I-JEPA is simple: from a single context block, predict the representations of various target blocks in the same image. A core design choice to guide I-JEPA towards producing semantic representations is the masking strategy; specifically, it is crucial to (a) sample target blocks with sufficiently large scale (semantic), and to (b) use a sufficiently informative (spatially distributed) context block. Empirically, when combined with Vision Transformers, we find I-JEPA to be highly scalable. For instance, we train a ViT-Huge/14 on ImageNet using 16 A100 GPUs in under 72 hours to achieve strong downstream performance across a wide range of tasks, from linear classification to object counting and depth prediction.
核心贡献 · Key contributions
提出 I-JEPA，一种非生成式自监督方法，从单个上下文块预测目标块表示。 Introduces I-JEPA, a non-generative self-supervised method predicting target block representations from a single context block.
表明在表示空间中进行预测比像素级重建能产生更具语义的特征。 Shows that predicting in representation space yields more semantic features than pixel-level reconstruction.
证明使用大目标块和信息性上下文的多块掩码策略对语义表示至关重要。 Demonstrates that a multi-block masking strategy with large target blocks and informative context is crucial for semantic representations.
无需手工数据增强，在线性分类、目标计数和深度预测等任务上实现强下游性能。 Achieves strong downstream performance across tasks like linear classification, object counting, and depth prediction without hand-crafted augmentations.
展示高可扩展性：在 ImageNet 上使用 16 块 A100 GPU 在 72 小时内训练 ViT-H/14，计算效率优于先前方法。 Shows high scalability: ViT-H/14 on ImageNet trained with 16 A100 GPUs in under 72 hours, outperforming prior methods in compute efficiency.
局限 · Limitations
评估主要在 ImageNet-1K 上进行，未充分探索对其他领域或模态的泛化。 Evaluation primarily on ImageNet-1K; generalization to other domains or modalities not fully explored.
需要仔细调整掩码策略（目标块尺度和上下文信息量）以达到最佳性能。 Requires careful tuning of masking strategy (target block scale and context informativeness) for optimal performance.
在某些设置下，语义任务性能可能仍落后于使用手工增强的视图不变方法。 Performance on semantic tasks may still lag behind view-invariance methods that use hand-crafted augmentations in some settings.
依赖指数移动平均目标编码器增加了复杂性和超参数敏感性。 Dependence on exponential moving average target encoder adds complexity and hyperparameter sensitivity.