Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.9% top-5 validation error (and 4.8% test error), exceeding the accuracy of human raters.
核心贡献 · Key contributions
提出批归一化以减少内部协变量偏移，从而支持更高的学习率和更快的训练。 Proposes Batch Normalization to reduce internal covariate shift, enabling higher learning rates and faster training.
利用小批量统计量归一化层输入，使归一化成为模型架构的一部分。 Normalizes layer inputs using mini-batch statistics, making normalization part of the model architecture.
通过防止饱和允许使用饱和非线性，并作为正则化器减少对 Dropout 的需求。 Allows use of saturating nonlinearities by preventing saturation, and acts as a regularizer reducing need for Dropout.
在 ImageNet 上以 14 倍少的训练步骤达到相同精度，并以 4.9%的 top-5 错误率创下新纪录。 Achieves same accuracy with 14x fewer training steps on ImageNet, and sets new state-of-the-art with 4.9% top-5 error.
引入可学习的仿射参数，在归一化后恢复网络的表示能力。 Introduces learned affine parameters to restore network representation capacity after normalization.
证明批归一化改善了梯度流动并稳定了参数增长。 Demonstrates that Batch Normalization improves gradient flow and stabilizes parameter growth.
局限 · Limitations
归一化依赖于小批量大小；过小的批量可能导致统计量不准确。 Normalization depends on mini-batch size; very small batches may lead to inaccurate statistics.
该方法假设层输入具有类高斯分布；非高斯输入可能降低效果。 The method assumes that layer inputs have Gaussian-like distribution; non-Gaussian inputs may reduce effectiveness.
批归一化在训练期间因计算每批统计量而增加计算开销。 Batch Normalization adds computational overhead during training due to per-batch statistics computation.
批归一化为何有效的理论理解仍不完整，尤其在梯度传播方面。 The theoretical understanding of why Batch Normalization works remains incomplete, especially for gradient propagation.
该方法可能不适用于循环网络或批量极小的任务。 The method may not generalize well to recurrent networks or tasks with very small batch sizes.
论文章节 · Sections（共 11）
摘要Abstract
1 引言1 Introduction
2 减少内部协变量偏移2 Towards Reducing Internal Covariate Shift
3 通过小批量统计进行归一化3 Normalization via Mini-Batch Statistics
3.1 批归一化网络的训练与推理3.1 Training and Inference with Batch-Normalized Networks