Text-to-image (T2I) models contain rich spatial priors. Synthesizing photorealistic, cluttered scenes requires an understanding of geometry, including perspective and relative scale. Prior works adapt T2I models to leverage this prior for depth prediction, but they require dense depth data and involve complex recipes. We propose Modality Forcing, a simple, scalable post-training recipe for joint image-depth generation using a single DiT trained on sparse depth data. Modality Forcing enables conditional and joint generation of image and depth in any permutation by assigning separate noise levels per modality. Per-modality decoders let us train on sparse, real-world depth and achieve strong, generalizable depth prediction. We further show that Modality Forcing inherits the scalability of T2I pre-training: by training a set of T2I models from scratch (370M to 3.3B parameters), we find that larger models trained on more image data produce more accurate depth. Our strongest model is competitive with state-of-the-art monocular depth estimators and reduces AbsRel by 57% relative to existing joint image-depth generative models. These results provide strong evidence that image generation is a scalable pre-training objective for spatial perception. https://modality-forcing.github.io/
核心贡献 · Key contributions
提出 Modality Forcing，一种简单的后训练方法，使用单个 DiT 在稀疏深度数据上训练，实现图像-深度联合生成。 Proposes Modality Forcing, a simple post-training recipe for joint image-depth generation using a single DiT trained on sparse depth data.
通过为每种模态分配独立的噪声水平，支持图像和深度任意排列的条件生成和联合生成。 Enables conditional and joint generation of image and depth in any permutation by assigning separate noise levels per modality.
通过从 3.7 亿到 33 亿参数的受控缩放实验，证明深度预测随 T2I 模型大小和训练数据规模扩展。 Demonstrates that depth prediction scales with T2I model size and training data size through controlled scaling experiments from 370M to 3.3B parameters.
实现最先进的单目深度估计，与现有联合图像-深度生成模型相比，AbsRel 降低 57%。 Achieves state-of-the-art monocular depth estimation, reducing AbsRel by 57% relative to existing joint image-depth generative models.
引入自蒸馏损失，在后训练期间保留 T2I 先验，防止灾难性遗忘。 Introduces self-distillation loss to preserve T2I prior during post-training, preventing catastrophic forgetting.
提供证据表明图像生成是空间感知的可扩展预训练目标。 Provides evidence that image generation is a scalable pre-training objective for spatial perception.
局限 · Limitations
缩放研究最多限于 33 亿参数，未推导完整的缩放定律。 Scaling study limited to at most 3.3B parameters and does not derive a full scaling law.
实验使用的深度样本相对较少，比 T2I 预训练数据少几个数量级。 Experiments use relatively few depth samples, several orders of magnitude fewer than T2I pretraining data.
架构调整可能进一步减少伪影或实现公制深度预测。 Architectural adjustments may further reduce artifacts or enable prediction of metric depth.
像素空间的深度标记化可能限制扩展到其他空间模态，如网格或点云。 Depth tokenization in pixel space may limit extension to other spatial modalities like meshes or point clouds.
在 ScanNet 上的性能因数据混合中包含训练集分割而受到干扰。 Performance on ScanNet is confounded by training split inclusion in data mixture.