When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition.
核心贡献 · Key contributions
提出 dropout 技术，在训练时随机丢弃一半特征检测器以防止协同适应。 Proposes dropout, a technique that randomly omits half of feature detectors during training to prevent co-adaptation.
表明 dropout 在小训练集上显著减少过拟合。 Shows dropout reduces overfitting significantly on small training sets.
展示 dropout 作为高效模型平均，近似指数级数量的网络。 Demonstrates dropout as efficient model averaging, approximating exponentially many networks.
在语音（TIMIT）和物体识别（CIFAR-10、ImageNet）基准上取得最优结果。 Achieves state-of-the-art results on speech (TIMIT) and object recognition (CIFAR-10, ImageNet) benchmarks.
引入每个隐藏单元的 L2 权重约束，允许大学习率和彻底的权重空间搜索。 Introduces L2 weight constraints per hidden unit to allow large learning rates and thorough weight-space search.
将 dropout 与进化理论联系，提出通过多种替代特征集实现鲁棒性。 Connects dropout to evolutionary theory, suggesting robustness through multiple alternative feature sets.
局限 · Limitations
全文使用 dropout 概率 0.5；最优概率可能因层或数据集而异。 Dropout probability of 0.5 is used throughout; optimal probability may vary per layer or dataset.
方法主要在前馈神经网络上测试；未探索对循环或其他架构的适用性。 Method primarily tested on feedforward neural networks; applicability to recurrent or other architectures not explored.
性能提升依赖于网络大小和训练数据量；在极大数据集上可能减弱。 Performance gains depend on network size and training data amount; may diminish with very large datasets.
测试时平均网络近似假设所有 dropout 网络重要性相等；可能不适用于所有情况。 Test-time mean network approximation assumes equal importance of all dropout networks; may not hold in all cases.
Dropout 因随机更新增加训练时间，且需仔细调整学习率和约束。 Dropout increases training time due to stochastic updates and requires careful tuning of learning rate and constraints.