分数蒸馏采样（SDS）通过蒸馏预训练的大规模文本到图像扩散模型，在文本到 3D 生成中显示出巨大潜力，但存在过饱和、过平滑和低多样性问题。在这项工作中，我们提出将 3D 参数建模为随机变量而非 SDS 中的常数，并提出了变分分数蒸馏（VSD），这是一个基于粒子的原则性变分框架，用于解释和解决文本到 3D 生成中的上述问题。我们证明 SDS 是 VSD 的一个特例，并且在 CFG 权重较小和较大时都会导致较差的样本。相比之下，VSD 能够像扩散模型中的祖先采样一样，在各种 CFG 权重下良好工作，并在使用常见 CFG 权重（如 7.5）时同时提高多样性和样本质量。我们进一步提出了文本到 3D 设计空间中的各种改进，例如蒸馏时间表和密度初始化，这些与蒸馏算法正交但尚未得到充分探索。我们的整体方法称为 ProlificDreamer，可以生成高渲染分辨率（即 512×512）和高保真度的 NeRF，具有丰富的结构和复杂效果（如烟雾和液滴）。此外，从 NeRF 初始化后，通过 VSD 微调的网格具有精细的细节和照片级真实感。项目页面和代码：https://ml.cs.tsinghua.edu.cn/prolificdreamer/
Score distillation sampling (SDS) has shown great promise in text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-saturation, over-smoothing, and low-diversity problems. In this work, we propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., $7.5$). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., $512\times512$) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic. Project page and codes: https://ml.cs.tsinghua.edu.cn/prolificdreamer/
核心贡献 · Key contributions
提出变分分数蒸馏（VSD），一种基于粒子的变分框架用于文本到 3D 生成。 Proposes Variational Score Distillation (VSD), a particle-based variational framework for text-to-3D generation.
证明 SDS 是 VSD 的特例，解释了过饱和、过平滑和低多样性问题。 Shows SDS is a special case of VSD, explaining over-saturation, over-smoothing, and low-diversity issues.
VSD 适用于正常 CFG 权重（如 7.5），同时提升多样性和样本质量。 VSD works with normal CFG weights (e.g., 7.5) and improves diversity and sample quality simultaneously.
引入高分辨率渲染（512x512）和退火时间调度以提升视觉质量。 Introduces high-resolution rendering (512x512) and annealed time schedule for improved visual quality.
提出场景初始化用于复杂场景生成，实现 360 度高保真 NeRF。 Proposes scene initialization for complex scene generation, enabling 360-degree high-fidelity NeRF.
通过从 NeRF 初始化的 VSD 微调，实现照片级真实感的纹理网格。 Achieves photo-realistic textured meshes via VSD fine-tuning from NeRF initialization.
局限 · Limitations
生成耗时数小时，远慢于扩散模型的图像生成。 Generation takes hours, much slower than image generation by diffusion models.
训练中的相机位姿与场景结构无关，限制了大型场景的细节。 Camera poses during training are independent of scene structure, limiting detail for large scenes.
复杂提示可能因基础 2D 模型表达能力有限而失败。 Complex prompts may fail due to limited expressiveness of the base 2D model.
由于视角相关提示的文本-图像对齐不佳，可能出现多面 Janus 问题。 Multi-face Janus problem may occur due to poor text-image alignment for view-dependent prompts.
内容创作和虚假内容生成可能带来有害社会影响。 Potential harmful social impacts from content creation and generation of fake content.
论文章节 · Sections（共 16）
摘要Abstract
1 引言1 Introduction
2 背景2 Background
3 变分分数蒸馏3 Variational Score Distillation
3.1 从 3D 分布采样作为变分推断3.1 Sampling from 3D Distribution as Variational Inference
3.2 变分分数蒸馏的更新规则3.2 Update Rule for Variational Score Distillation
3.3 与 SDS 的比较3.3 Comparison with SDS
4 ProlificDreamer4 ProlificDreamer
4.1 文本到 3D 生成的设计空间4.1 Design Space of Text-to-3D Generation
4.2 3D 表示与训练4.2 3D Representation and Training
5 实验5 Experiments
5.1 ProlificDreamer 的结果5.1 Results of ProlificDreamer