Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute a mean and variance which are then used to normalize the summed input to that neuron on each training case. This significantly reduces the training time in feed-forward neural networks. However, the effect of batch normalization is dependent on the mini-batch size and it is not obvious how to apply it to recurrent neural networks. In this paper, we transpose batch normalization into layer normalization by computing the mean and variance used for normalization from all of the summed inputs to the neurons in a layer on a single training case. Like batch normalization, we also give each neuron its own adaptive bias and gain which are applied after the normalization but before the non-linearity. Unlike batch normalization, layer normalization performs exactly the same computation at training and test times. It is also straightforward to apply to recurrent neural networks by computing the normalization statistics separately at each time step. Layer normalization is very effective at stabilizing the hidden state dynamics in recurrent networks. Empirically, we show that layer normalization can substantially reduce the training time compared with previously published techniques.
核心贡献 · Key contributions
提出层归一化，对单个训练样本的层内所有求和输入计算均值和方差。 Proposes layer normalization, computing mean and variance from all summed inputs in a layer per training case.
层归一化通过在每个时间步分别计算归一化统计量，适用于循环神经网络。 Layer normalization works for RNNs by computing normalization statistics separately at each time step.
层归一化稳定循环网络中的隐藏状态动态，减少训练时间。 Layer normalization stabilizes hidden state dynamics in recurrent networks, reducing training time.
实验表明在问答和语言建模等多个任务上收敛更快且泛化更好。 Empirically shows faster convergence and better generalization on multiple tasks including QA and language modeling.
层归一化对整体权重矩阵的缩放和每个样本的特征平移具有不变性。 Layer normalization is invariant to re-scaling of entire weight matrix and per-case feature shifting.
层归一化对小批量大小和在线学习设置具有鲁棒性。 Layer normalization is robust to small mini-batch sizes and online learning settings.
局限 · Limitations
层归一化在卷积神经网络中表现不如批归一化。 Layer normalization underperforms batch normalization in convolutional neural networks.
该方法假设层内隐藏单元贡献相似，这在 CNN 中可能不成立。 The method assumes hidden units in a layer contribute similarly, which may not hold for CNNs.
与批归一化不同，层归一化对单个权重向量的缩放不具有不变性。 Layer normalization is not invariant to individual weight vector scaling, unlike batch normalization.
理论分析局限于广义线性模型，可能无法完全捕捉深度网络行为。 Theoretical analysis is limited to generalized linear models and may not fully capture deep network behavior.
实验评估主要针对 RNN，前馈和 CNN 的结果不够全面。 Empirical evaluation focuses on RNNs; feed-forward and CNN results are less comprehensive.