We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Dropout, the most successful technique for regularizing neural networks, does not work well with RNNs and LSTMs. In this paper, we show how to correctly apply dropout to LSTMs, and show that it substantially reduces overfitting on a variety of tasks. These tasks include language modeling, speech recognition, image caption generation, and machine translation.
核心贡献 · Key contributions
展示了如何通过将 dropout 限制在非循环连接上来正确应用于 LSTM。 Shows how to correctly apply dropout to LSTMs by restricting it to non-recurrent connections.
在语言建模、语音识别、图像描述生成和机器翻译等多个任务上，证明了 LSTM 过拟合的显著减少。 Demonstrates substantial reduction of overfitting in LSTMs across language modeling, speech recognition, image captioning, and machine translation.
提供了经验证据，表明对非循环连接使用 dropout 保留了 LSTM 的长期记忆能力。 Provides empirical evidence that dropout on non-recurrent connections preserves LSTM's long-term memory ability.
在 Penn Tree Bank 语言建模任务上，使用大型 LSTM 达到了最先进的困惑度。 Achieves state-of-the-art perplexity on Penn Tree Bank language modeling with a large LSTM.
提高了语音识别中的帧准确率和机器翻译中的 BLEU 分数。 Improves frame accuracy in speech recognition and BLEU score in machine translation.
表明在图像描述生成中，dropout 可以使单个 LSTM 模型达到与集成模型相当的效果。 Shows that dropout can make a single LSTM model as effective as an ensemble in image captioning.
局限 · Limitations
dropout 仅应用于 LSTM，未测试其他 RNN 变体。 Dropout is applied only to LSTM; other RNN variants are not tested.
最佳 dropout 概率因任务而异，需要调参。 Optimal dropout probability varies across tasks and requires tuning.
在图像描述生成中，该方法与集成结合时未提升性能。 Method does not improve performance when combined with ensemble in image captioning.
实验仅限于特定数据集，可能无法推广到所有领域。 Experiments are limited to specific datasets and may not generalize to all domains.
关于 dropout 为何在非循环连接上有效的理论分析有限。 Theoretical analysis of why dropout works on non-recurrent connections is limited.
论文章节 · Sections（共 13）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related work
3 使用 LSTM 单元正则化 RNN3 Regularizing RNNs with LSTM cells