尽管近期关于文本条件 3D 物体生成的工作取得了有希望的结果，但最先进的方法通常需要多个 GPU 小时才能生成单个样本。这与最先进的生成式图像模型形成鲜明对比，后者在几秒或几分钟内就能生成样本。在本文中，我们探索了一种替代的 3D 物体生成方法，该方法在单个 GPU 上仅需 1-2 分钟即可生成 3D 模型。我们的方法首先使用文本到图像扩散模型生成单个合成视图，然后使用第二个扩散模型基于生成的图像生成 3D 点云。尽管我们的方法在样本质量上仍不及最先进水平，但其采样速度快一到两个数量级，为某些用例提供了实用的权衡。我们在 https://github.com/openai/point-e 发布了预训练的点云扩散模型以及评估代码和模型。
While recent work on text-conditional 3D object generation has shown promising results, the state-of-the-art methods typically require multiple GPU-hours to produce a single sample. This is in stark contrast to state-of-the-art generative image models, which produce samples in a number of seconds or minutes. In this paper, we explore an alternative method for 3D object generation which produces 3D models in only 1-2 minutes on a single GPU. Our method first generates a single synthetic view using a text-to-image diffusion model, and then produces a 3D point cloud using a second diffusion model which conditions on the generated image. While our method still falls short of the state-of-the-art in terms of sample quality, it is one to two orders of magnitude faster to sample from, offering a practical trade-off for some use cases. We release our pre-trained point cloud diffusion models, as well as evaluation code and models, at https://github.com/openai/point-e.
核心贡献 · Key contributions
提出两阶段扩散流水线用于文本到 3D 生成，将采样时间缩短至单 GPU 1-2 分钟。 Proposes a two-stage diffusion pipeline for text-to-3D generation, reducing sampling time to 1-2 minutes on a single GPU.
引入基于 Transformer 的扩散模型，用于生成以图像为条件的彩色点云。 Introduces a Transformer-based diffusion model for generating colored point clouds conditioned on images.
证明以 CLIP 图像潜在网格为条件优于单嵌入条件。 Demonstrates that conditioning on a grid of CLIP image latents outperforms single-embedding conditioning.
展示模型规模扩展可提升 P-FID 收敛速度和 CLIP R-Precision。 Shows scaling model size improves P-FID convergence and CLIP R-Precision.
发布预训练点云扩散模型和评估代码以促进可复现性。 Releases pre-trained point cloud diffusion models and evaluation code for reproducibility.
为文本到 3D 合成提供实用的速度-质量权衡，支持快速原型设计。 Provides a practical speed-quality trade-off for text-to-3D synthesis, enabling rapid prototyping.
局限 · Limitations
样本质量不及 DreamFusion 等基于优化的最先进方法。 Sample quality falls short of state-of-the-art optimization-based methods like DreamFusion.
输出为低分辨率点云（4K 点），缺乏精细形状和纹理。 Outputs are low-resolution point clouds (4K points), lacking fine-grained shape and texture.