通过估计数据分布的梯度进行生成建模

Generative Modeling by Estimating Gradients of the Data Distribution

宋飏 Yang Song · Stanford · 2019-07-12 · arXiv:1907.05600 ↗ · 被引 5692

摘要 · Abstract

我们引入了一种新的生成模型，其中样本通过朗之万动力学生成，利用分数匹配估计的数据分布梯度。由于当数据位于低维流形上时，梯度可能定义不良且难以估计，我们使用不同级别的高斯噪声扰动数据，并联合估计相应的分数，即所有噪声级别下扰动数据分布的梯度向量场。对于采样，我们提出了一种退火朗之万动力学，其中随着采样过程接近数据流形，我们使用对应逐渐减小噪声级别的梯度。我们的框架允许灵活的模型架构，训练期间无需采样或使用对抗方法，并提供了一个可用于原则性模型比较的学习目标。我们的模型在 MNIST、CelebA 和 CIFAR-10 数据集上生成的样本可与 GAN 相媲美，在 CIFAR-10 上实现了 8.87 的新最优初始分数。此外，我们通过图像修复实验证明了我们的模型学习了有效的表示。

We introduce a new generative model where samples are produced via Langevin dynamics using gradients of the data distribution estimated with score matching. Because gradients can be ill-defined and hard to estimate when the data resides on low-dimensional manifolds, we perturb the data with different levels of Gaussian noise, and jointly estimate the corresponding scores, i.e., the vector fields of gradients of the perturbed data distribution for all noise levels. For sampling, we propose an annealed Langevin dynamics where we use gradients corresponding to gradually decreasing noise levels as the sampling process gets closer to the data manifold. Our framework allows flexible model architectures, requires no sampling during training or the use of adversarial methods, and provides a learning objective that can be used for principled model comparisons. Our models produce samples comparable to GANs on MNIST, CelebA and CIFAR-10 datasets, achieving a new state-of-the-art inception score of 8.87 on CIFAR-10. Additionally, we demonstrate that our models learn effective representations via image inpainting experiments.

核心贡献 · Key contributions

提出基于分数的生成建模，利用分数匹配和朗之万动力学生成样本。
Proposes score-based generative modeling using Langevin dynamics with score matching.
引入噪声条件分数网络（NCSN），联合估计多个噪声水平下的分数。
Introduces Noise Conditional Score Networks (NCSN) to estimate scores at multiple noise levels.
开发退火朗之万动力学采样方法，逐步降低噪声水平。
Develops annealed Langevin dynamics for sampling, gradually reducing noise levels.
在 CIFAR-10 上实现无条件生成的最新最佳起始分数 8.87。
Achieves state-of-the-art inception score of 8.87 on CIFAR-10 for unconditional generation.
展示无需特殊架构即可有效进行图像修复。
Demonstrates effective image inpainting without specialized architectures.
提供可处理的训练目标，用于原则性模型比较。
Provides a tractable objective for principled model comparisons.

局限 · Limitations

需要多个噪声水平和退火采样，增加计算成本。
Requires multiple noise levels and annealed sampling, increasing computational cost.
尽管有噪声扰动，低密度区域的分数估计仍可能不准确。
Score estimation may be inaccurate in low-density regions despite noise perturbation.
对于高度多模态分布，朗之万动力学混合可能较慢。
Langevin dynamics mixing can be slow for highly multimodal distributions.
模型性能依赖于噪声水平和步长的仔细调整。
Model performance depends on careful tuning of noise levels and step sizes.
主要在图像数据上评估，未探索对其他领域的泛化。
Primarily evaluated on image data; generalization to other domains is not explored.

论文章节 · Sections（共 16）

摘要 Abstract
1 引言 1 Introduction
2 基于分数的生成建模 2 Score-based generative modeling
2.1 分数匹配用于分数估计 2.1 Score matching for score estimation
2.2 朗之万动力学采样 2.2 Sampling with Langevin dynamics
3 基于分数的生成建模的挑战 3 Challenges of score-based generative modeling
3.1 流形假设 3.1 The manifold hypothesis
3.2 低数据密度区域 3.2 Low data density regions
4 噪声条件分数网络：学习与推理 4 Noise Conditional Score Networks: learning and inference
4.1 噪声条件分数网络 4.1 Noise Conditional Score Networks
4.2 通过分数匹配学习 NCSN 4.2 Learning NCSNs via score matching
4.3 通过退火朗之万动力学进行 NCSN 推理 4.3 NCSN inference via annealed Langevin dynamics
5 实验 5 Experiments
6 相关工作 6 Related work
7 结论 7 Conclusion
致谢 Acknowledgements

阅读逐段中英对照全文 →