We present weight normalization: a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction. By reparameterizing the weights in this way we improve the conditioning of the optimization problem and we speed up convergence of stochastic gradient descent. Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also be applied successfully to recurrent models such as LSTMs and to noise-sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited. Although our method is much simpler, it still provides much of the speed-up of full batch normalization. In addition, the computational overhead of our method is lower, permitting more optimization steps to be taken in the same amount of time. We demonstrate the usefulness of our method on applications in supervised image recognition, generative modelling, and deep reinforcement learning.
核心贡献 · Key contributions
提出权重归一化，一种将权重向量长度与方向解耦的重参数化方法，以改善优化问题的条件。 Proposes weight normalization, a reparameterization decoupling weight vector length from direction to improve optimization conditioning.
在监督学习、生成建模和强化学习任务中，展示了权重归一化加速随机梯度下降收敛的效果。 Demonstrates faster convergence of stochastic gradient descent with weight normalization across supervised, generative, and reinforcement learning tasks.
表明权重归一化在 LSTM 等循环模型及批归一化不适用的噪声敏感应用中表现良好。 Shows weight normalization works well with recurrent models like LSTMs and noise-sensitive applications where batch normalization fails.
引入仅均值批归一化与权重归一化的组合，在不使用数据增强的 CIFAR-10 上达到最优测试误差。 Introduces mean-only batch normalization combined with weight normalization, achieving state-of-the-art CIFAR-10 test error without data augmentation.
提供理论分析，表明权重归一化将梯度投影到当前权重向量之外，降低噪声并稳定学习。 Provides theoretical analysis showing weight normalization projects gradients away from current weight vector, reducing noise and stabilizing learning.
提供一种计算廉价且确定性的批归一化替代方案，开销更低且无小批量依赖。 Offers a computationally cheap and deterministic alternative to batch normalization with lower overhead and no minibatch dependencies.
局限 · Limitations
权重归一化不固定跨层特征尺度，需要基于数据的初始化以获得最佳性能。 Weight normalization does not fix feature scale across layers, requiring careful data-based initialization for optimal performance.
该方法可能不适用于无法进行基于数据初始化的模型，例如缺乏明确小批量结构的循环网络。 The method may not be applicable to models where data-based initialization is infeasible, such as recurrent networks without clear minibatch structure.
实验结果表明，批归一化在早期每轮进度略优，但权重归一化以更低计算成本弥补。 Empirical results show batch normalization achieves slightly better per-epoch progress early on, though weight normalization compensates with lower computational cost.
梯度投影分析假设无动量或自适应学习率，可能不适用于 Adam 等现代优化器。 The analysis of gradient projection assumes no momentum or adaptive learning rates, which may not hold for modern optimizers like Adam.
仅均值批归一化与权重归一化组合引入额外复杂性，可能不适用于所有架构。 Mean-only batch normalization combined with weight normalization introduces additional complexity and may not generalize to all architectures.
论文章节 · Sections（共 13）
摘要Abstract
1 引言1 Introduction
2 权重归一化2 Weight Normalization
2.1 梯度2.1 Gradients
2.2 与批归一化的关系2.2 Relation to batch normalization
3 数据相关的参数初始化3 Data-Dependent Initialization of Parameters