我们提出了 Stable Video 3D（SV3D）——一种潜在视频扩散模型，用于从单张图像生成围绕 3D 物体的高分辨率、多视角轨道视频。最近的 3D 生成工作提出了将 2D 生成模型适配到新视角合成（NVS）和 3D 优化的技术。然而，这些方法由于视角有限或 NVS 不一致而存在若干缺点，从而影响 3D 物体生成的性能。在这项工作中，我们提出了 SV3D，它将图像到视频扩散模型适配到新颖多视图合成和 3D 生成，从而利用了视频模型的泛化能力和多视图一致性，同时进一步为 NVS 添加了显式相机控制。我们还提出了改进的 3D 优化技术，以使用 SV3D 及其 NVS 输出进行图像到 3D 生成。在多个数据集上的大量实验结果（包括 2D 和 3D 指标以及用户研究）表明，与先前工作相比，SV3D 在 NVS 和 3D 重建方面达到了最先进的性能。
We present Stable Video 3D (SV3D) -- a latent video diffusion model for high-resolution, image-to-multi-view generation of orbital videos around a 3D object. Recent work on 3D generation propose techniques to adapt 2D generative models for novel view synthesis (NVS) and 3D optimization. However, these methods have several disadvantages due to either limited views or inconsistent NVS, thereby affecting the performance of 3D object generation. In this work, we propose SV3D that adapts image-to-video diffusion model for novel multi-view synthesis and 3D generation, thereby leveraging the generalization and multi-view consistency of the video models, while further adding explicit camera control for NVS. We also propose improved 3D optimization techniques to use SV3D and its NVS outputs for image-to-3D generation. Extensive experimental results on multiple datasets with 2D and 3D metrics as well as user study demonstrate SV3D's state-of-the-art performance on NVS as well as 3D reconstruction compared to prior works.
核心贡献 · Key contributions
提出 SV3D，一种用于从单张图像进行新颖多视图合成和 3D 生成的潜在视频扩散模型。 Proposes SV3D, a latent video diffusion model for novel multi-view synthesis and 3D generation from a single image.
将图像到视频扩散模型与显式相机姿态控制相结合，实现多视图一致的新颖视图合成。 Adapts image-to-video diffusion model with explicit camera pose control for multi-view consistent novel view synthesis.
引入三角形无分类器引导缩放，提升轨道视频质量。 Introduces triangle classifier-free guidance scaling for improved orbital video quality.
提出解耦照明模型和掩码分数蒸馏采样损失，用于高质量 3D 生成。 Proposes disentangled illumination model and masked score distillation sampling loss for high-quality 3D generation.
在多个数据集上的新颖视图合成和 3D 重建中达到最先进性能。 Achieves state-of-the-art performance on novel view synthesis and 3D reconstruction across multiple datasets.
在真实世界图像上展示强泛化能力，并在用户研究中获得偏好。 Demonstrates strong generalization to real-world images and user preference in human studies.
局限 · Limitations
SV3D 仅处理相机控制的 2 个自由度（仰角和方位角）。 SV3D only handles 2 degrees of freedom (elevation and azimuth) for camera control.
镜面反射表面的视图不一致，给 3D 重建带来挑战。 View inconsistency for mirror-like reflective surfaces, challenging 3D reconstruction.
朗伯着色模型无法表示反射表面。 Lambertian shading model cannot represent reflective surfaces.
训练需要大量计算资源（4 节点 8 块 A100 GPU，6 天）。 Requires large computational resources for training (4 nodes of 8 A100 GPUs for 6 days).
SDS 损失若简单应用可能导致训练不稳定和伪影。 SDS loss can cause unstable training and artifacts if applied naively.
论文章节 · Sections（共 10）
摘要Abstract
1 引言1 Introduction
2.1 新视角合成2.1 Novel View Synthesis
2.2 三维生成2.2 3D Generation
3 SV3D：新颖的多视角合成3 SV3D: Novel Multi-view Synthesis
3.1 实验与结果3.1 Experiments and Results
4 利用 SV3D 从单张图像进行三维生成4 3D Generation from a Single Image Using SV3D
4.1 三维优化策略与损失函数4.1 3D Optimization Strategies and Losses