Self-Normalizing Neural Networks
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→深度学习通过卷积神经网络（CNN）和循环神经网络（RNN）在视觉和自然语言处理领域引发了革命。然而，标准前馈神经网络（FNN）在深度学习中的成功案例却很少。表现良好的 FNN 通常较浅，因此无法利用多层次的抽象表示。我们引入了自归一化神经网络（SNN）以实现高级抽象表示。批归一化需要显式归一化，而 SNN 的神经元激活会自动收敛到零均值和单位方差。SNN 的激活函数是“缩放指数线性单元”（SELU），它具有自归一化特性。利用巴拿赫不动点定理，我们证明了接近零均值和单位方差的激活值在通过多个网络层传播时，即使在噪声和扰动存在的情况下，也会收敛到零均值和单位方差。SNN 的这种收敛特性允许（1）训练深层网络，（2）使用强正则化，（3）使学习高度鲁棒。此外，对于不接近单位方差的激活值，我们证明了方差的上界和下界，因此梯度消失和爆炸是不可能的。我们在（a）UCI 机器学习库的 121 个任务、（b）药物发现基准测试和（c）天文学任务上，将 SNN 与标准 FNN 以及其他机器学习方法（如随机森林和支持向量机）进行了比较。SNN 在 121 个 UCI 任务上显著优于所有竞争的 FNN 方法，在 Tox21 数据集上优于所有竞争方法，并在一个天文学数据集上创造了新纪录。获胜的 SNN 架构通常非常深。实现代码可在 github.com/bioinf-jku/SNNs 获取。
Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -- even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs and other machine learning methods such as random forests and support vector machines. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: github.com/bioinf-jku/SNNs.