生成对抗网络（GAN）是强大的生成模型，但面临训练不稳定的问题。近期提出的 Wasserstein GAN（WGAN）在稳定 GAN 训练方面取得了进展，但有时仍只能生成低质量样本或无法收敛。我们发现这些问题通常源于 WGAN 中为了对判别器施加 Lipschitz 约束而使用的权重裁剪，这可能导致不良行为。我们提出一种替代权重裁剪的方法：惩罚判别器关于其输入的梯度范数。我们提出的方法优于标准 WGAN，并且能够在几乎不需要超参数调整的情况下稳定训练多种 GAN 架构，包括 101 层 ResNet 和离散数据上的语言模型。此外，我们在 CIFAR-10 和 LSUN 卧室数据集上生成了高质量图像。
Generative Adversarial Networks (GANs) are powerful generative models, but suffer from training instability. The recently proposed Wasserstein GAN (WGAN) makes progress toward stable training of GANs, but sometimes can still generate only low-quality samples or fail to converge. We find that these problems are often due to the use of weight clipping in WGAN to enforce a Lipschitz constraint on the critic, which can lead to undesired behavior. We propose an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input. Our proposed method performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning, including 101-layer ResNets and language models over discrete data. We also achieve high quality generations on CIFAR-10 and LSUN bedrooms.
核心贡献 · Key contributions
指出 WGAN 中的权重裁剪会导致梯度爆炸/消失以及无法捕获高阶矩。 Identifies weight clipping in WGAN causes gradient explosion/vanishing and failure to capture high-order moments.
提出梯度惩罚来强制 Lipschitz 约束，替代权重裁剪以实现稳定训练。 Proposes gradient penalty to enforce Lipschitz constraint, replacing weight clipping for stable training.
实现多种 GAN 架构的稳定训练，包括 101 层残差网络和语言模型，几乎无需调参。 Enables stable training of diverse GAN architectures, including 101-layer ResNets and language models, with minimal tuning.
在 CIFAR-10 上取得最优 Inception 分数，在 LSUN 卧室数据集上生成高质量图像。 Achieves state-of-the-art Inception scores on CIFAR-10 and high-quality generation on LSUN bedrooms.
成功建模离散分布（字符级语言），无需离散采样。 Successfully models discrete distributions (character-level language) without discrete sampling.
局限 · Limitations
梯度惩罚增加了计算梯度范数的开销，延长训练时间。 Gradient penalty adds computational overhead for gradient norm calculation, increasing training time.
理论保证仅适用于平滑激活函数；ReLU 在实践中有效，但在零测点处梯度未定义。 Theoretical guarantee only for smooth activations; ReLU works empirically but gradients are undefined at measure-zero points.
惩罚系数 λ 在所有实验中固定为 10，最优值可能因架构而异。 Penalty coefficient λ is fixed at 10 across experiments; optimal values may vary by architecture.
批评者仍会在数据有限（如 MNIST 子集）时过拟合，降低与样本质量的相关性。 Critic can still overfit on limited data (e.g., MNIST subset), reducing correlation with sample quality.
使用 ELU 激活函数时训练失败，需要改用 softplus 等平滑非线性函数。 Training fails with ELU activations; requires smooth nonlinearities like softplus-based alternatives.
论文章节 · Sections（共 13）
摘要Abstract
2.1 生成对抗网络2.1 Generative adversarial networks
最优 WGAN 判别器的性质2.3 Properties of the optimal WGAN critic
3 权重约束的难点3 Difficulties with weight constraints
25 个高斯分布25 Gaussians
3.1 能力利用不足3.1 Capacity underuse
3.2 梯度爆炸与消失3.2 Exploding and vanishing gradients
5 实验5 Experiments
5.2 在 LSUN bedrooms 数据集上训练多种架构5.2 Training varied architectures on LSUN bedrooms
5.3 相较于权重裁剪的性能提升5.3 Improved performance over weight clipping
5.4 CIFAR-10 和 LSUN 卧室上的样本质量5.4 Sample quality on CIFAR-10 and LSUN bedrooms
5.5 使用连续生成器对离散数据建模5.5 Modeling discrete data with a continuous generator
5.6 有意义的损失曲线与过拟合检测5.6 Meaningful loss curves and detecting overfitting