长短期记忆

长短期记忆 Long Short-Term Memory Sepp Hochreiter

通过循环反向传播学习在延长的时间间隔内存储信息需要很长时间，主要是因为误差反向流动不足且衰减。我们简要回顾了 Hochreiter（1991）对这一问题的分析，然后通过引入一种新颖、高效的基于梯度的方法——长短期记忆（LSTM）来解决它。在不会造成损害的地方截断梯度，LSTM 可以通过在特殊单元内强制恒定误差流经恒定误差传送带，学习桥接超过 1000 个离散时间步的最小时间延迟。乘法门单元学习打开和关闭对恒定误差流的访问。LSTM 在空间和时间上是局部的；其每时间步和权重的计算复杂度为 O(1)。我们在人工数据上的实验涉及局部、分布式、实值和带噪声的模式表示。与实时循环学习、时间反向传播、循环级联相关、Elman 网络和神经序列分块相比，LSTM 实现了更多成功的运行，并且学习速度更快。LSTM 还解决了先前循环网络算法从未解决过的复杂、人工长时延任务。1 引言

Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We briefly review Hochreiter's (1991) analysis of this problem, then address it by introducing a novel, efficient, gradient-based method called long short-term memory (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O(1). Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with real-time recurrent learning, backpropagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long-time-lag tasks that have never been solved by previous recurrent network algorithms. 1 Introduction

原则上，循环网络可以利用其反馈连接以激活的形式存储近期输入事件的表示（短期记忆，与由缓慢变化的权重体现的长期记忆相对）。这对许多应用具有潜在重要意义，包括语音处理、非马尔可夫控制和音乐作曲（Mozer, 1992）。然而，最广泛使用的学习短期记忆内容的算法需要太多时间或根本效果不佳，特别是当输入与相应教师信号之间的最小时间延迟较长时。尽管理论上有趣，现有方法并未提供比具有有限时间窗口的前馈网络中的反向传播更明确的实际优势。本文回顾了对此问题的分析并提出了补救措施。

In principle, recurrent networks can use their feedback connections to store representations of recent input events in the form of activations (short-term memory, as opposed to long-term memory embodied by slowly changing weights). This is potentially significant for many applications, including speech processing, non-Markovian control, and music composition (Mozer, 1992). The most widely used algorithms for learning what to put in short-term memory, however, take too much time or do not work well at all, especially when minimal time lags between inputs and corresponding teacher signals are long. Although theoretically fascinating, existing methods do not provide clear practical advantages over, say, backpropagation in feed-forward nets with limited time windows. This article reviews an analysis of the problem and suggests a remedy.

问题。使用传统的随时间反向传播（BPTT; Williams & Zipser, 1992; Werbos, 1988）或实时循环学习（RTRL; Robinson & Fallside, 1987），在时间上反向流动的误差信号往往（1）爆炸或（2）消失；反向传播误差的时间演化指数依赖于权重的大小（Hochreiter, 1991）。情况 1 可能导致权重的振荡；情况 2 中，学习桥接长时间延迟需要过多的时间或根本不起作用（见第 3 节）。本文提出了长短期记忆（LSTM），一种新颖的循环网络架构，结合了适当的基于梯度的学习算法。LSTM 旨在克服这些误差反向流动问题。它能够学习桥接超过 1000 步的时间间隔，即使在有噪声、不可压缩的输入序列情况下，也不会损失短时延能力。这是通过一种高效、基于梯度的算法实现的，该算法针对一种架构强制恒定（因此既不爆炸也不消失）误差流经特殊单元的内部状态（前提是梯度计算在特定架构点截断；但这不影响长期误差流）。第 2 节简要回顾先前的工作。第 3 节首先概述了 Hochreiter（1991）对消失误差的详细分析。然后以教学目的介绍了恒定误差反向传播的朴素方法，并强调了其在信息存储和检索方面的问题。这些问题导致了第 4 节描述的 LSTM 架构。第 5 节介绍大量实验以及与竞争方法的比较。LSTM 优于它们，并学会了解决其他循环网络算法未解决的复杂人工任务。第 6 节讨论 LSTM 的局限性和优势。附录包含算法的详细描述（A.1）和显式误差流公式（A.2）。

The problem. With conventional backpropagation through time (BPTT; Williams & Zipser, 1992; Werbos, 1988) or real-time recurrent learning (RTRL; Robinson & Fallside, 1987), error signals flowing backward in time tend to (1) blow up or (2) vanish; the temporal evolution of the backpropagated error exponentially depends on the size of the weights (Hochreiter, 1991). Case 1 may lead to oscillating weights; in case 2, learning to bridge long time lags takes a prohibitive amount of time or does not work at all (see section 3). This article presents long short-term memory (LSTM), a novel recurrent network architecture in conjunction with an appropriate gradient-based learning algorithm. LSTM is designed to overcome these error backflow problems. It can learn to bridge time intervals in excess of 1000 steps even in case of noisy, incompressible input sequences, without loss of short-time-lag capabilities. This is achieved by an efficient, gradient-based algorithm for an architecture enforcing constant (thus, neither exploding nor vanishing) error flow through internal states of special units (provided the gradient computation is truncated at certain architecture-specific points; this does not affect long-term error flow, though). Section 2 briefly reviews previous work. Section 3 begins with an outline of the detailed analysis of vanishing errors due to Hochreiter (1991). It then introduces a naive approach to constant error backpropagation for didactic purposes and highlights its problems concerning information storage and retrieval. These problems lead to the LSTM architecture described in section 4. Section 5 presents numerous experiments and comparisons with competing methods. LSTM outperforms them and also learns to solve complex, artificial tasks no other recurrent net algorithm has solved. Section 6 discusses LSTM's limitations and advantages. The appendix contains a detailed description of the algorithm (A.1) and explicit error flow formulas (A.2).

本节关注具有时变输入的循环网络（与具有固定输入和基于不动点的梯度计算的网络相对；例如，Almeida, 1987; Pineda, 1987）。2.1 梯度下降变体。Elman（1988）、Fahlman（1991）、Williams（1989）、Schmidhuber（1992a）、Pearlmutter（1989）以及 Pearlmutter 综合概述（1995）中的许多相关算法的方法，都面临着与 BPTT 和 RTRL 相同的问题（见第 1 节和第 3 节）。

This section focuses on recurrent nets with time-varying inputs (as opposed to nets with stationary inputs and fixed-point-based gradient calculations; e.g., Almeida, 1987; Pineda, 1987). 2.1 Gradient-Descent Variants. The approaches of Elman (1988), Fahlman (1991), Williams (1989), Schmidhuber (1992a), Pearlmutter (1989), and many of the related algorithms in Pearlmutter's comprehensive overview (1995) suffer from the same problems as BPTT and RTRL (see sections 1 and 3).

Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We briefly review Hochreiter’s (1991) analysis of this problem, then address it by introducing a novel, efficient, gradient-based method called long short-term memory (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O(1). Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In compar-isons with real-time recurrent learning, back propagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunk-ing, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long-time-lag tasks that have never been solved by previous recurrent network algorithms. 1 Introduction

2.2 时间延迟。其他仅对短时间滞后有效的方法包括时延神经网络（Lang, Waibel, & Hinton, 1990）和 Plate 的方法（Plate, 1993），该方法基于旧激活的加权和更新单元激活（另见 de Vries & Principe, 1991）。Lin 等人（1996）提出了时延网络的变体，称为 NARX 网络。

2.2 Time Delays. Other methods that seem practical for short time lags only are time-delay neural networks (Lang, Waibel, & Hinton, 1990) and Plate’s method (Plate, 1993), which updates unit activations based on a weighted sum of old activations (see also de Vries & Principe, 1991). Lin et al. (1996) propose variants of time-delay networks called NARX networks.

2.3 时间常数。为了处理长时间滞后，Mozer（1992）使用影响单元激活变化的时间常数（deVries 和 Principe 1991 年的方法实际上可以看作是时延神经网络和时间常数的混合）。然而，对于长时间滞后，时间常数需要外部微调（Mozer, 1992）。Sun、Chen 和 Lee（1993）的替代方法通过将旧激活与（缩放的）当前净输入相加来更新循环单元的激活。然而，净输入往往会扰动存储的信息，这使得长期存储不切实际。

2.3 Time Constants. To deal with long time lags, Mozer (1992) uses time constants influencing changes of unit activations (deVries and Principe’s 1991 approach may in fact be viewed as a mixture of time-delay neural networks and time constants). For long time lags, however, the time constants need external fine tuning (Mozer, 1992). Sun, Chen, and Lee’s alternative approach (1993) updates the activation of a recurrent unit by adding the old activation and the (scaled) current net input. The net input, however, tends to perturb the stored information, which makes long-term storage impractical.

2.4 Ring 的方法。Ring（1993）也提出了一种桥接长时间滞后的方法。每当他的网络中的单元接收到冲突的误差信号时，他就添加一个高阶单元来影响相应的连接。尽管他的方法有时速度非常快，但要桥接涉及 100 步的时间滞后可能需要添加 100 个单元。此外，Ring 的网络不能推广到未见过的滞后持续时间。

2.4 Ring’s Approach. Ring (1993) also proposed a method for bridging long time lags. Whenever a unit in his network receives conflicting error signals, he adds a higher-order unit influencing appropriate connections. Although his approach can sometimes be extremely fast, to bridge a time lag involving 100 steps may require the addition of 100 units. Also, Ring’s net does not generalize to unseen lag durations.

2.5 Bengio 等人的方法。Bengio、Simard 和 Frasconi（1994）研究了模拟退火、多网格随机搜索、时间加权伪牛顿优化和离散误差传播等方法。他们的“锁存”和“双序列”问题与本文中最小时间滞后为 100 的问题 3a 非常相似（见实验 3）。Bengio 和 Frasconi（1994）还提出了一种用于传播目标的期望最大化方法。通过 n 个所谓的状态网络，在给定时间，他们的系统只能处于 n 个不同状态之一。（另见第 5 节开头。）但要解决诸如加法问题（第 5.4 节）之类的连续问题，他们的系统将需要不可接受数量的状态（即状态网络）。

2.5 Bengio et al.’s Approach. Bengio, Simard, and Frasconi (1994) investigate methods such as simulated annealing, multigrid random search, time-weighted pseudo-Newton optimization, and discrete error propagation. Their “latch” and “two-sequence” problems are very similar to problem 3a in this article with minimal time lag 100 (see Experiment 3). Bengio and Frasconi (1994) also propose an expectation-maximization approach for propagating targets. With n so-called state networks, at a given time, their system can be in one of only n different states. (See also the beginning of section 5.) But to solve continuous problems such as the adding problem (section 5.4), their system would require an unacceptable number of states (i.e., state networks).

2.6 卡尔曼滤波器。Puskorius 和 Feldkamp（1994）使用卡尔曼滤波技术来提高循环网络的性能。由于他们使用了“一个导数折扣因子，用于指数衰减过去动态导数的影响”，没有理由相信他们的卡尔曼滤波器训练的循环网络对于极长的最小时间滞后是有用的。

2.6 Kalman Filters. Puskorius and Feldkamp (1994) use Kalman filter techniques to improve recurrent net performance. Since they use “a derivative discount factor imposed to decay exponentially the effects of past dynamic derivatives,” there is no reason to believe that their Kalman filter-trained recurrent networks will be useful for very long minimal time lags.

2.7 二阶网络。我们将看到 LSTM 使用乘法单元（MUs）来保护误差流免受不必要的干扰。然而，它并不是第一个使用 MUs 的循环网络方法。例如，Watrous 和 Kuhn (1992) 在二阶网络中使用了 MUs。与 LSTM 的一些区别：(1) Watrous 和 Kuhn 的架构不强制恒定误差流，且不设计用于解决长时间滞后问题；(2) 它具有全连接的二阶 sigma-pi 单元，而 LSTM 架构的 MUs 仅用于门控访问恒定误差流；(3) Watrous 和 Kuhn 的算法每个时间步需要\(O(W^2)\)次操作，而我们的只需\(O(W)\)，其中\(W\)是权重数量。另见 Miller 和 Giles (1993) 关于 MUs 的进一步工作。

2.7 Second Order Nets. We will see that LSTM uses multiplicative units (MUs) to protect error flow from unwanted perturbations. It is not the first recurrent net method using MUs, though. For instance, Watrous and Kuhn (1992) use MUs in second-order nets. There are some differences from LSTM: (1) Watrous and Kuhn’s architecture does not enforce constant error flow and is not designed to solve long-time-lag problems; (2) it has fully connected second-order sigma-pi units, while the LSTM architecture’s MUs are used only to gate access to constant error flow; and (3) Watrous and Kuhn’s algorithm costs \(O(W^2)\) operations per time step, ours only \(O(W)\), where \(W\) is the number of weights. See also Miller and Giles (1993) for additional work on MUs.

2.8 简单权重猜测。为了避免基于梯度的方法的长时间滞后问题，我们可以简单地随机初始化所有网络权重，直到得到的网络恰好正确分类所有训练序列。事实上，我们最近发现（Schmidhuber & Hochreiter, 1996; Hochreiter & Schmidhuber, 1996, 1997），简单的权重猜测解决了许多 Bengio 等人(1994)、Bengio 和 Frasconi(1994)、Miller 和 Giles(1993)以及 Lin 等人(1996)中的问题，并且比这些作者提出的算法更快。这并不意味着权重猜测是一个好算法。这只是说明这些问题非常简单。更现实的任务需要大量自由参数（例如输入权重）或高权重精度（例如连续值参数），使得猜测完全不可行。

2.8 Simple Weight Guessing. To avoid long-time-lag problems of gradient-based approaches, we may simply randomly initialize all network weights until the resulting net happens to classify all training sequences correctly. In fact, recently we discovered (Schmidhuber & Hochreiter, 1996; Hochreiter & Schmidhuber, 1996, 1997) that simple weight guessing solves many of the problems in Bengio et al. (1994), Bengio and Frasconi (1994), Miller and Giles (1993), and Lin et al. (1996) faster than the algorithms these authors proposed. This does not mean that weight guessing is a good algorithm. It just means that the problems are very simple. More realistic tasks require either many free parameters (e.g., input weights) or high weight precision (e.g., for continuous-valued parameters), such that guessing becomes completely infeasible.

2.9 自适应序列分块器。Schmidhuber 的分层分块系统（1992b, 1993）确实有能力跨越任意时间滞后，但只有当跨子序列存在局部可预测性时才行（参见 Mozer, 1992）。例如，在他的博士后论文中，Schmidhuber（1993）使用分层循环网络快速解决了某些语法学习任务，这些任务涉及超过 1000 步的最小时间滞后。然而，随着噪声水平增加和输入序列变得不易压缩，分块系统的性能会下降。LSTM 没有这个问题。

2.9 Adaptive Sequence Chunkers. Schmidhuber’s hierarchical chunker systems (1992b, 1993) do have a capability to bridge arbitrary time lags, but only if there is local predictability across the subsequences causing the time lags (see also Mozer, 1992). For instance, in his postdoctoral thesis, Schmidhuber (1993) uses hierarchical recurrent nets to solve rapidly certain grammar learning tasks involving minimal time lags in excess of 1000 steps. The performance of chunker systems, however, deteriorates as the noise level increases and the input sequences become less compressible. LSTM does not suffer from this problem.

3 恒定误差反向传播 3.1 指数衰减误差

3 Constant Error Backpropagation 3.1 Exponentially Decaying Error

3.1.1 传统的 BPTT（例如 Williams & Zipser, 1992）。在时间\(t\)时刻输出单元\(k\)的目标记为\(d_k(t)\)。使用均方误差，\(k\)的误差信号为

3.1.1 Conventional BPTT (e.g., Williams & Zipser, 1992). Output unit \(k\)’s target at time \(t\) is denoted by \(d_k(t)\). Using mean squared error, \(k\)’s error signal is

是非输入单元 \(i\) 的激活，该单元具有可微激活函数

is the activation of a noninput unit \(i\) with differentiable activation function

\(w_{ij} y_j(t-1)\) 是单元 \(i\) 的当前净输入，\(w_{ij}\) 是从单元 \(j\) 到 \(i\) 的连接权重。某个非输出单元 \(j\) 的反向传播误差信号为

\(w_{ij} y_j(t-1)\) is unit \(i\)'s current net input, and \(w_{ij}\) is the weight on the connection from unit \(j\) to \(i\). Some nonoutput unit \(j\)'s backpropagated error signal is

对 \(w_{jl}\) 总权重更新的相应贡献为 \(α ϑ_j(t) y_l(t-1)\)，其中 \(α\) 是学习率，\(l\) 表示连接到单元 \(j\) 的任意单元。

The corresponding contribution to \(w_{jl}\)'s total weight update is \(α ϑ_j(t) y_l(t-1)\), where \(α\) is the learning rate and \(l\) stands for an arbitrary unit connected to unit \(j\).

3.1.2 霍克赖特分析概述（1991，第 19–21 页）。假设我们有一个全连接网络，其非输入单元索引从 1 到 \(n\)。让我们关注从单元 \(u\) 到单元 \(v\) 的局部误差流（稍后我们将看到该分析立即扩展到全局误差流）。在时间步 \(t\) 发生在任意单元 \(u\) 的误差被向后传播 \(q\) 个时间步，到达任意单元 \(v\)。这将误差缩放以下因子：

3.1.2 Outline of Hochreiter's Analysis (1991, pp. 19–21). Suppose we have a fully connected net whose noninput unit indices range from 1 to \(n\). Let us focus on local error flow from unit \(u\) to unit \(v\) (later we will see that the analysis immediately extends to global error flow). The error occurring at an arbitrary unit \(u\) at time step \(t\) is propagated back in time for \(q\) time steps, to an arbitrary unit \(v\). This will scale the error by the following factor:

\(w_{l_1 l_0} w_{l_2 l_1} \ldots w_{l_q l_{q-1}} > 1\) (3.1) 令 \(l_q = v\) 且 \(l_0 = u\)，我们得到：

\(w_{l_1 l_0} w_{l_2 l_1} \ldots w_{l_q l_{q-1}} > 1\) (3.1) With \(l_q = v\) and \(l_0 = u\), we obtain:

\( (net_{l_m}(t-m)) w_{l_m l_{m-1}} \ (3.2) (归纳证明)。这 \(n^{q-1}\) 项的和 \( \prod_{m=1}^{q} f'(net_{l_m}(t-m)) w_{l_m l_{m-1}} \)

\( (net_{l_m}(t-m)) w_{l_m l_{m-1}} \ (3.2) (proof by induction). The sum of the \(n^{q-1}\) terms \( \prod_{m=1}^{q} f'(net_{l_m}(t-m)) w_{l_m l_{m-1}} \)

\( (net_{l_m}(t-m)) w_{l_m l_{m-1}} \ 决定了总误差反向传播（注意，由于求和项可能有不同符号，增加单元数 n 并不一定会增加误差流）。

\( (net_{l_m}(t-m)) w_{l_m l_{m-1}} \ determines the total error backflow (note that since the summation terms may have different signs, increasing the number of units n does not necessarily increase error flow).

3.1.3 方程(3.2)的直观解释。如果

3.1.3 Intuitive Explanation of Equation 3.2. If

\( (net_{l_m}(t-m)) w_{l_m l_{m-1}} \ > 1.0 对所有 m 成立（例如，当 \(f_{l_m}\) 为线性时可能发生），则最大乘积随 q 呈指数增长。即，误差爆炸，到达单元 v 的冲突误差信号可能导致权重振荡和学习不稳定（关于误差爆炸或分叉，另见 Pineda, 1988; Baldi & Pineda, 1991; Doya, 1992）。另一方面，如果

\( (net_{l_m}(t-m)) w_{l_m l_{m-1}} \ > 1.0 for all m (as can happen, e.g., with linear \(f_{l_m}\)), then the largest product increases exponentially with q. That is, the error blows up, and conflicting error signals arriving at unit v can lead to oscillating weights and unstable learning (for error blowups or bifurcations, see also Pineda, 1988; Baldi & Pineda, 1991; Doya, 1992). On the other hand, if

(net l m (t − m)) w l m lm−1 (3.2) (proof by induction). The sum of the n q−1 terms ∏qm=1 f ′

是 0.25。如果 \(y_l^{m-1}\) 是常数且不等于零，那么 \(| f'(net_l^m) w_{lm} w_{l m-1} |\) 在 \(|w_{l m} w_{l m-1}| \rightarrow \infty\) 时趋于零，在 \(|w_{l m} w_{l m-1}| < 4.0\) 时小于 1.0（例如，如果绝对值最大权重值 \(w_{\max}\) 小于 4.0）。因此，对于传统的逻辑斯谛 Sigmoid 激活函数，只要权重的绝对值小于 4.0，误差流就容易消失，尤其是在训练初期。一般而言，使用更大的初始权重也无济于事，因为如上所述，当 \(|w_{l m} w_{l m-1}| \rightarrow \infty\) 时，相关导数趋于零的速度比绝对值权重增长“更快”（而且，一些权重必须通过零来改变符号）。同样，增加学习率也没有用；它不会改变长程误差流与短程误差流的比例。BPTT 对近期的干扰过于敏感。（Bengio 等人（1994）提出了非常类似的、更新的分析。）

is 0.25. If \(y_l^{m-1}\) is constant and not equal to zero, then \(| f'(net_l^m) w_{lm} w_{l m-1} |\) takes on maximal values where it goes to zero for \(|w_{l m} w_{l m-1}| \rightarrow \infty\), and is less than 1.0 for \(|w_{l m} w_{l m-1}| < 4.0\) (e.g., if the absolute maximal weight value \(w_{\max}\) is smaller than 4.0). Hence with conventional logistic sigmoid activation functions, the error flow tends to vanish as long as the weights have absolute values below 4.0, especially in the beginning of the training phase. In general, the use of larger initial weights will not help, though, as seen above, for \(|w_{l m} w_{l m-1}| \rightarrow \infty\) the relevant derivative goes to zero "faster" than the absolute weight can grow (also, some weights will have to change their signs by crossing zero). Likewise, increasing the learning rate does not help either; it will not change the ratio of long-range error flow and short-range error flow. BPTT is too sensitive to recent distractions. (A very similar, more recent analysis was presented by Bengio et al., 1994.)

3.1.4 全局误差流。上面的局部误差流分析立即表明全局误差流也会消失。为了看清这一点，计算

3.1.4 Global Error Flow. The local error flow analysis above immediately shows that global error flow vanishes too. To see this, compute

3.1.5 缩放因子的弱上界。以下稍微扩展的误差消失分析也考虑了单元数 n。对于 \(q > 1\)，方程 3.2 可以重写为

3.1.5 Weak Upper Bound for Scaling Factor. The following, slightly extended vanishing error analysis also takes n, the number of units, into account. For \(q > 1\), equation 3.2 can be rewritten as

is 0.25. If y l m−1 is constant and not equal to zero, then | f ′

(net l m )wlm l m−1 | takes on maximal values where

其中权重矩阵 W 定义为 \([W]_{ij} := w_{ij}\)，v 的传出权重向量 \(W_v\) 定义为 \([W_v]_i := [W]_{iv} = w_{iv}\)，u 的传入权重向量

where the weight matrix W is defined by \([W]_{ij} := w_{ij}\), v's outgoing weight vector \(W_v\) is defined by \([W_v]_i := [W]_{iv} = w_{iv}\), u's incoming weight vector

\(W_u^T\) 定义为 \([W_u^T]_i := [W]_{ui} = w_{ui}\)，且对于 \(m = 1, \ldots, q\)，\(F'(t - m)\) 是一阶导数的对角矩阵，定义为 \([F'(t - m)]_{ij} := 0\) 如果

\(W_u^T\) is defined by \([W_u^T]_i := [W]_{ui} = w_{ui}\), and for \(m = 1, \ldots, q\), \(F'(t - m)\) is the diagonal matrix of first-order derivatives defined as \([F'(t - m)]_{ij} := 0\) if

否则为 \(\text{net}_i(t-m)\)。这里 \(T\) 是转置算子，\([A]_{ij}\) 是矩阵 A 第 i 列第 j 行的元素，\([x]_i\) 是向量 x 的第 i 个分量。使用与向量范数 \(\|\cdot\|_x\) 相容的矩阵范数 \(\|\cdot\|_A\)，我们定义

\(\text{net}_i(t-m)\) otherwise. Here \(T\) is the transposition operator, \([A]_{ij}\) is the element in the ith column and jth row of matrix A, and \([x]_i\) is the ith component of vector x. Using a matrix norm \(\|\cdot\|_A\) compatible with vector norm \(\|\cdot\|_x\), we define

对于 \(\max_{i=1,\ldots,n} \{|x_i|\} \leq \|x\|_x\) 我们得到 \(|x^T y| \leq n \|x\|_x \|y\|_x\)。由于

For \(\max_{i=1,\ldots,n} \{|x_i|\} \leq \|x\|_x\) we get \(|x^T y| \leq n \|x\|_x \|y\|_x\). Since

我们得到以下不等式：

we obtain the following inequality:

\(\lVert W_u^T \rVert_{x} = \lVert W^T e_u \rVert_{x} \leq \lVert W \rVert_{A} \lVert e_u \rVert_{x} \leq \lVert W \rVert_{A},\)

\(\lVert W_u^T \rVert_{x} = \lVert W^T e_u \rVert_{x} \leq \lVert W \rVert_{A} \lVert e_u \rVert_{x} \leq \lVert W \rVert_{A},\)

其中 e_k 是第 k 个分量为 1、其余分量为 0 的单位向量。注意，这是一个弱极值情况的上界；只有当所有 \(\lVert F'(t-m) \rVert_{A}\) 都取最大值，并且从单元 u 到单元的所有误差反向传播路径的贡献具有相同的符号时，该上界才会达到。

where e_k is the unit vector whose components are 0 except for the k-th component, which is 1. Note that this is a weak, extreme case upper bound; it will be reached only if all \(\lVert F'(t-m) \rVert_{A}\) take on maximal values, and if the contributions of all paths across which error flows back from unit u to unit

v 具有相同的符号。然而，大的 \(\lVert W \rVert_{A}\) 通常会导致 \(\lVert F'(t-m) \rVert_{A}\) 值较小，实验证实了这一点（例如，参见 Hochreiter, 1991）。例如，对于范数

v have the same sign. Large \(\lVert W \rVert_{A}\), however, typically results in small values of \(\lVert F'(t-m) \rVert_{A}\), as confirmed by experiments (see, e.g., Hochreiter, 1991). For example, with norms

对于逻辑斯谛 Sigmoid 函数，该值为 0.25。我们观察到，如果

= 0.25 for the logistic sigmoid. We observe that if

那么 \(\lVert W \rVert_{A} \leq n w_{\max} < 4.0\) 将导致指数衰减。通过设置

then \(\lVert W \rVert_{A} \leq n w_{\max} < 4.0\) will result in exponential decay. By setting

更多结果请参见 Hochreiter (1991)。

We refer to Hochreiter (1991) for additional results.

3.2.1 单个单元。为了避免误差信号消失，我们如何通过单个与自身连接的单元\(j\)实现恒定误差流？根据上述规则，在时刻\(t\)，\(j\)的局部误差反向流动为\(\vartheta_j(t) = f'(\text{net}_j(t)) \vartheta_j(t+1) w_{jj}\)。为了强制\(j\)的误差流恒定，我们要求\(f'(\text{net}_j(t)) w_{jj} = 1.0\)。注意到与 Mozer (1992)的固定时间常数系统的相似性——1.0 的时间常数适用于潜在无限的时间滞后。

3.2.1 A Single Unit. To avoid vanishing error signals, how can we achieve constant error flow through a single unit \(j\) with a single connection to itself? According to the rules above, at time \(t\), \(j\)'s local error backflow is \(\vartheta_j(t) = f'(\text{net}_j(t)) \vartheta_j(t+1) w_{jj}\). To enforce constant error flow through \(j\), we require \(f'(\text{net}_j(t)) w_{jj} = 1.0\). Note the similarity to Mozer's fixed time constant system (1992)—a time constant of 1.0 is appropriate for potentially infinite time lags.

3.2.2 恒定误差传送带。对上述微分方程积分，我们得到

3.2.2 The Constant Error Carousel. Integrating the differential equation above, we obtain

We refer to Hochreiter (1991) for additional results.

3.2.1 A Single Unit. To avoid vanishing error signals, how can we achieve constant error flow through a single unit j with a single connec-tion to itself? According to the rules above, at time t, j’s local error backflow is ϑj(t) = f ′

对于任意的 net_j(t)。这意味着 f_j 必须是线性的，且单元 j 的激活必须保持恒定：

for arbitrary net_j(t). This means f_j has to be linear, and unit j's activation has to remain constant:

y_j(t+1) = f_j(\text{net}_j(t+1)) = f_j(w_{jj} y_j(t)) = y_j(t)。

y_j(t+1) = f_j(\text{net}_j(t+1)) = f_j(w_{jj} y_j(t)) = y_j(t).

在实验中，通过使用恒等函数 f_j: f_j(x) = x, ∀x，并设置 w_{jj} = 1.0 来确保这一点。我们称之为恒定误差传送带（CEC）。CEC 将是 LSTM 的核心特征（见第 4 节）。当然，单元 j 不仅会连接到自身，还会连接到其他单元。这引发了两个明显的相关问题（也存在于所有其他基于梯度的方法中）：1. 输入权重冲突：为简单起见，让我们专注于一个额外的输入权重 w_{ji}。假设通过响应某个输入而开启单元 j 并使其长时间保持活跃（直到它帮助计算所需的输出），可以减少总误差。假设

In the experiments, this will be ensured by using the identity function f_j: f_j(x) = x, ∀x, and by setting w_{jj} = 1.0. We refer to this as the constant error carousel (CEC). CEC will be LSTM's central feature (see section 4). Of course, unit j will not only be connected to itself but also to other units. This invokes two obvious, related problems (also inherent in all other gradient-based approaches): 1. Input weight conflict: For simplicity, let us focus on a single additional input weight w_{ji}. Assume that the total error can be reduced by switching on unit j in response to a certain input and keeping it active for a long time (until it helps to compute a desired output). Provided

i 非零，由于相同的传入权重必须同时用于存储某些输入和忽略其他输入，在此期间 w_{ji} 经常会收到冲突的权重更新信号（回想 j 是线性的）。这些信号会试图使 w_{ji} 参与 (1) 存储输入（通过开启 j）和 (2) 保护输入（通过防止 j

i is nonzero, since the same incoming weight has to be used for both storing certain inputs and ignoring others, w_{ji} will often receive conflicting weight update signals during this time (recall that j is linear). These signals will attempt to make w_{ji} participate in (1) storing the input (by switching on j) and (2) protecting the input (by preventing j

for arbitrary net j(t). This means fj has to be linear, and unit j’s activation has to remain constant:

来自于被不相关的后续输入关闭）。这个冲突使得学习变得困难，并需要一种更具上下文敏感性的机制来通过输入权重控制写操作。2. 输出权重冲突：假设 j 被激活并当前存储了某些之前的输入。为简单起见，让我们专注于一个额外的输出权重 \(w_{kj}\)。同一个 \(w_{kj}\) 必须同时用于在某些时候检索 j 的内容，并在其他时候防止 j 干扰 k。只要单元 j 非零，\(w_{kj}\) 就会吸引序列处理过程中产生的冲突权重更新信号。这些信号将试图让 \(w_{kj}\) 参与访问存储在 j 中的信息，并在不同时间保护单元 k 免受 j 的扰动。例如，在许多任务中，存在某些短期滞后的误差，可以在早期训练阶段减少。然而，（我们不使用 Pearlmutter (1995) 那种微分意义上的“时间常数”表达式）在后期训练阶段，j 可能会突然开始导致那些本已看似可控的情况中出现可避免的误差，因为它试图参与减少更困难的长期滞后误差。再次，这个冲突使得学习变得困难，并需要一种更具上下文敏感性的机制来通过输出权重控制读操作。当然，输入和输出权重冲突并非长期滞后所特有；它们也出现在短期滞后中。然而，它们的影响在长期滞后情况下变得尤为显著。随着时间滞后的增加，存储的信息必须被保护免受更长时间段的扰动，尤其是在学习的高级阶段，越来越多的已经正确的输出也需要保护免受扰动。由于上述问题，简单方法除了在某些涉及局部输入输出表示和非重复输入模式的简单问题（见 Hochreiter, 1991; Silva, Amarel, Langlois, & Almeida, 1996）之外，效果并不好。下一节将展示如何正确实现。

from being switched off by irrelevant later inputs). This conflict makes learning difficult and calls for a more context-sensitive mechanism for controlling write operations through input weights. 2. Output weight conflict: Assume j is switched on and currently stores some previous input. For simplicity, let us focus on a single additional outgoing weight \(w_{kj}\). The same \(w_{kj}\) has to be used for both retrieving j’s content at certain times and preventing j from disturbing k at other times. As long as unit j is nonzero, \(w_{kj}\) will attract conflicting weight update signals generated during sequence processing. These signals will attempt to make \(w_{kj}\) participate in accessing the information stored in j and—at different times—protecting unit k from being perturbed by j. For instance, with many tasks there are certain short-time-lag errors that can be reduced in early training stages. However, (We do not use the expression “time constant” in the differential sense, as Pearlmutter (1995) does) at later training stages, j may suddenly start to cause avoidable errors in situations that already seemed under control by attempting to participate in reducing more difficult long-time-lag errors. Again, this conflict makes learning difficult and calls for a more context-sensitive mechanism for controlling read operations through output weights. Of course, input and output weight conflicts are not specific for long time lags; they occur for short time lags as well. Their effects, however, become particularly pronounced in the long-time-lag case. As the time lag increases, stored information must be protected against perturbation for longer and longer periods, and, especially in advanced stages of learning, more and more already correct outputs also require protection against perturbation. Due to the problems set out, the naive approach does not work well except in the case of certain simple problems involving local input-output representations and nonrepeating input patterns (see Hochreiter, 1991; Silva, Amarel, Langlois, & Almeida, 1996). The next section shows how to do it right.

4 长短期记忆的概念 4.1 记忆单元与门单元。为了构建一种允许误差通过特殊的自连接单元恒定流动的架构，同时避免简单方法的缺点，我们扩展了第 3.2 节中由自连接线性单元 j 体现的 CEC，引入了额外的特征。引入了一个乘法输入门单元来保护存储在 j 中的记忆内容免受不相关输入的扰动，并引入了一个乘法输出门单元来保护其他单元免受当前存储在 j 中的不相关记忆内容的扰动。由此产生的更复杂的单元称为记忆单元（见图 1）。第 j 个记忆单元表示为 \(c_j\)。每个记忆单元围绕一个具有固定自连接（CEC）的中心线性单元构建。除了 \(net_{c_j}\) 之外，\(c_j\) 还从乘法单元 \(out_j\)（输出门）和另一个乘法单元 \(in_j\)（输入门）接收输入。\(in_j\) 在时间 t 的激活记为 \(y^{in}_j(t)\)，\(out_j\) 的激活记为 \(y^{out}_j(t)\)。我们有

4 The Concept of Long Short-Term Memory 4.1 Memory Cells and Gate Units. To construct an architecture that allows for constant error flow through special, self-connected units without the disadvantages of the naive approach, we extend the CEC embodied by the self-connected, linear unit j from section 3.2 by introducing additional features. A multiplicative input gate unit is introduced to protect the memory contents stored in j from perturbation by irrelevant inputs, and a multiplicative output gate unit is introduced to protect other units from perturbation by currently irrelevant memory contents stored in j. The resulting, more complex unit is called a memory cell (see Figure 1). The jth memory cell is denoted \(c_j\). Each memory cell is built around a central linear unit with a fixed self-connection (the CEC). In addition to \(net_{c_j}\), \(c_j\) gets input from a multiplicative unit \(out_j\) (the output gate), and from another multiplicative unit \(in_j\) (the input gate). \(in_j\)’s activation at time t is denoted by \(y^{in}_j(t)\), \(out_j\)’s by \(y^{out}_j(t)\). We have

\(y^{out}_j(t) = f_{out_j}(net^{out}_j(t)); y^{in}_j(t) = f_{in_j}(net^{in}_j(t))\)

\(y^{out}_j(t) = f_{out_j}(net^{out}_j(t)); y^{in}_j(t) = f_{in_j}(net^{in}_j(t))\)

\(w_{in_j}^u y_u(t-1)\)。

\(w_{in_j}^u y_u(t-1)\).

记忆细胞架构与门控单元 g h1.0 net wy in y out net cg y in = g+s c s c y in h y out net w cin out w i c cjjjjout wjin jjjj jj yjjjjii i

图 1：记忆细胞 c_j（方框）及其门控单元 in_j、out_j 的架构。权重为 1.0 的自循环连接表示一个时间步的延迟反馈。它构成了 CEC（恒定误差传送带）的基础。门控单元开启和关闭对 CEC 的访问。详情见正文和附录 A.1。我们还提供

Figure 1: Architecture of memory cell c_j (the box) and its gate units in_j, out_j. The self-recurrent connection (with weight 1.0) indicates feedback with a delay of one time step. It builds the basis of the CEC. The gate units open and close access to CEC. See text and appendix A.1 for details. We also have

求和索引 u 可以代表输入单元、门控单元、记忆细胞，甚至传统的隐藏单元（如果有的话，见第 4.3 节）。所有这些不同类型的单元都可能传递有关网络当前状态的有用信息。例如，输入门（输出门）可以利用来自其他记忆细胞的输入来决定是否在其记忆细胞中存储（访问）某些信息。甚至可能存在像 w_{c_j c_j} 这样的循环自连接。网络拓扑由用户定义。示例见图 2。在时刻 t，c_j 的输出 y_{c_j}(t) 的计算公式为

The summation indices u may stand for input units, gate units, memory cells, or even conventional hidden units if there are any (see section 4.3). All these different types of units may convey useful information about the current state of the net. For instance, an input gate (output gate) may use inputs from other memory cells to decide whether to store (access) certain information in its memory cell. There even may be recurrent self-connections like w_{c_j c_j}. It is up to the user to define the network topology. See Figure 2 for an example. At time t, c_j's output y_{c_j}(t) is computed as

\(s_{c_j}(0) = 0, s_{c_j}(t) = s_{c_j}(t-1) + y_{in_j}(t) g(net_{c_j}(t)), \quad t>0\)

\(s_{c_j}(0) = 0, s_{c_j}(t) = s_{c_j}(t-1) + y_{in_j}(t) g(net_{c_j}(t)), \quad t>0\)

可微函数 g 对 net_{c_j} 进行压缩；可微函数 h

The differentiable function g squashes net_{c_j}; the differentiable function h

根据内部状态 s_{c_j} 缩放记忆细胞输出。

scales memory cell outputs computed from the internal state s_{c_j}.

4.2 为什么使用门单元？为了避免输入权重冲突，输入门 in_j 控制流向记忆细胞 c_j 的输入连接 ⟀(w_{c_j i}⟀) 的错误流。为了避免 c_j 的输出权重冲突，输出门 out_j 控制来自单元 j 输出的错误流（长短期记忆）。

4.2 Why Gate Units? To avoid input weight conflicts, in j controls the error flow to memory cell cj's input connections ⟀(w_{c_j i}⟀). To circumvent cj's output weight conflicts, out j controls the error flow from unit j's output Long Short-Term Memory.

输出 隐藏 输入 out1 in1 out2 in2 1 细胞块 块 1 细胞块 块 2 细胞 2 细胞 2

output hidden input out 1 in 1 out 2 in 2 1 cell block block 1 cell block block 2 cell 2 cell 2

图 2：一个具有 8 个输入单元、4 个输出单元和两个大小为 2 的记忆细胞块的网络示例。in1 标记输入门，out1 标记输出门，cell1/block1 标记块 1 的第一个记忆细胞。cell1/block1 的架构与图 1 中的相同，带有门单元 in1 和 out1（注意，将图 1 逆时针旋转 90 度，将与图 2 的相应部分匹配）。该示例假设密集连接：每个门单元和每个记忆细胞都看到所有非输出单元。但为简单起见，每层只显示一种类型单元的输出权重。采用高效的截断更新规则，错误流仅通过连接到输出单元的连接以及细胞块内的固定自连接（此处未显示；见图 1）流动。错误流一旦“想要”离开记忆细胞或门单元就被截断。因此，上面没有显示任何连接用于将错误传播回连接起源的单元（除了到输出单元的连接），尽管连接本身是可修改的。这就是为什么截断的 LSTM 算法如此高效，尽管它能够桥接非常长的时间滞后。详见正文和附录。图 2 显示了实验 6a 中使用的架构；仅省略了非输入单元的偏置。换句话说，网络可以使用 ⟀(in_j⟀) 决定何时保留或覆盖记忆细胞 ⟀(c_j⟀) 中的信息，以及使用 ⟀(out_j⟀) 决定何时访问记忆细胞 ⟀(c_j⟀) 以及何时防止其他单元被 ⟀(c_j⟀) 干扰。

Figure 2: Example of a net with eight input units, four output units, and two memory cell blocks of size 2. in1 marks the input gate, out1 marks the output gate, and cell1/block1 marks the first memory cell of block1. cell1/block1's architecture is identical to the one in Figure 1, with gate units in1 and out1 (note that by rotating Figure 1 by 90 degrees anticlockwise, it will match with the corresponding parts of Figure 2). The example assumes dense connectivity: each gate unit and each memory cell sees all non-output units. For simplicity, however, outgoing weights of only one type of unit are shown for each layer. With the efficient, truncated update rule, error flows only through connections to output unit, and through fixed self-connections within cell blocks (not shown here; see Figure 1). Error flow is truncated once it "wants" to leave memory cells or gate units. Therefore, no connection shown above serves to propagate error back to the unit from which the connection originates (except for connections to output units), although the connections themselves are modifiable. That is why the truncated LSTM algorithm is so efficient, despite its ability to bridge very long time lags. See the text and the appendix for details. Figure 2 shows the architecture used for experiment 6a; only the bias of the non-input units is omitted. In other words, the net can use ⟀(in_j⟀) to decide when to keep or override information in memory cell ⟀(c_j⟀) and ⟀(out_j⟀) to decide when to access memory cell ⟀(c_j⟀) and when to prevent other units from being perturbed by ⟀(c_j⟀).

（见图 1）。困在记忆细胞 CEC 中的误差信号无法改变，但是通过输出门在不同时间流入细胞的误差信号可能会叠加。输出门必须学会通过适当地缩放误差来捕获其 CEC 中的哪些误差。输入门必须学会何时释放误差，同样通过适当地缩放。本质上，乘法门单元打开和关闭对通过 CEC 的恒定误差流的访问。分布式输出表示通常需要输出门。但两种门类型并非总是必要的；一个可能就足够了。例如，在第 5 节的实验 2a 和 2b 中，可以只使用输入门。事实上，在局部输出编码的情况下不需要输出门；通过简单地将相应的权重设置为零，可以防止记忆细胞干扰已经学好的输出。然而，即使在这种情况下，输出门也是有益的：它们可以防止网络存储长时间滞后记忆（通常难以学习）的尝试干扰代表易于学习的短时间滞后记忆的激活。（例如，这在实验 1 中将非常有用。）

(see Figure 1). Error signals trapped within a memory cell's CEC cannot change, but different error signals flowing into the cell (at different times) via its output gate may get superimposed. The output gate will have to learn which errors to trap in its CEC by appropriately scaling them. The input gate will have to learn when to release errors, again by appropriately scaling them. Essentially the multiplicative gate units open and close access to constant error flow through CEC. Distributed output representations typically do require output gates. Both gate types are not always necessary, though; one may be sufficient. For instance, in experiments 2a and 2b in section 5, it will be possible to use input gates only. In fact, output gates are not required in case of local output encoding; preventing memory cells from perturbing already learned outputs can be done by simply setting the corresponding weights to zero. Even in this case, however, output gates can be beneficial: they prevent the net's attempts at storing long-time-lag memories (which are usually hard to learn) from perturbing activations representing easily learnable short-time-lag memories. (This will prove quite useful in experiment 1, for instance.)

4.3 网络拓扑。我们使用具有一个输入层、一个隐藏层和一个输出层的网络。完全自连接的隐藏层包含记忆细胞和相应的门单元（为方便起见，我们将记忆细胞和门单元都视为位于隐藏层）。隐藏层还可能包含向门单元和记忆细胞提供输入的常规隐藏单元。所有层中除门单元外的所有单元都向上一层（或所有更高层）的所有单元具有有向连接（作为输入）；参见实验 2a 和 2b。

4.3 Network Topology. We use networks with one input layer, one hidden layer, and one output layer. The (fully) self-connected hidden layer contains memory cells and corresponding gate units (for convenience, we refer to both memory cells and gate units as being located in the hidden layer). The hidden layer may also contain conventional hidden units providing inputs to gate units and memory cells. All units (except for gate units) in all layers have directed connections (serve as inputs) to all units in the layer above (or to all higher layers; see experiments 2a and 2b).

4.4 记忆单元块。共享同一输入门和同一输出门的 S 个记忆单元构成一个称为记忆单元块的结构，其大小为

4.4 Memory Cell Blocks. S memory cells sharing the same input gate and the same output gate form a structure called a memory cell block of size

S。记忆单元块有利于信息存储。与传统的神经网络一样，在单个单元内编码分布式输入并不容易。由于每个记忆单元块具有与单个记忆单元相同数量的门控单元（即两个），因此块结构甚至可以略微更高效。大小为 1 的记忆单元块只是一个简单的记忆单元。在第 5 节的实验中，我们将使用各种大小的记忆单元块。

S. Memory cell blocks facilitate information storage. As with conventional neural nets, it is not so easy to code a distributed input within a single cell. Since each memory cell block has as many gate units as a single memory cell (namely, two), the block architecture can be even slightly more efficient. A memory cell block of size 1 is just a simple memory cell. In the experiments in section 5, we will use memory cell blocks of various sizes.

4.5 学习。我们使用 RTRL 的一个变体（例如，Robinson & Fallside, 1987），该变体考虑了由输入门和输出门引起的乘法动力学改变。为了确保通过记忆单元内部状态的误差反向传播不衰减，与截断 BPTT（例如，Williams & Peng, 1990）一样，到达记忆单元净输入（对于单元 c_j，这包括

4.5 Learning. We use a variant of RTRL (e.g., Robinson & Fallside, 1987) that takes into account the altered, multiplicative dynamics caused by input and output gates. To ensure nondecaying error backpropagation through internal states of memory cells, as with truncated BPTT (e.g., Williams & Peng, 1990), errors arriving at memory cell net inputs (for cell c_j, this includes

\(net_{c_j}, net_{in_j}, net_{out_j}\)不会在时间上进一步反向传播（尽管它们确实会改变输入权重）。只有在记忆单元内部，误差才会通过先前的内部状态\(s_{c_j}\)进行反向传播。为了理解这一点，一旦误差信号到达记忆单元输出，它会受到输出门激活值和\(h'\)的缩放。然后它进入记忆单元的 CEC，在该处它可以无限地回流而永远不会被缩放。当它通过输入门和\(g'\)离开记忆单元时，它会再次受到输入门激活值和\(g'\)的缩放。在截断之前，它用于改变输入权重（公式见附录）。

\(net_{c_j}, net_{in_j}, net_{out_j}\) do not get propagated back further in time (although they do serve to change the incoming weights). Only within memory cells, are errors propagated back through previous internal states \(s_{c_j}\). To visualize this, once an error signal arrives at a memory cell output, it gets scaled by output gate activation and \(h'\). Then it is within the memory cell's CEC, where it can flow back indefinitely without ever being scaled. When it leaves the memory cell through the input gate and \(g'\), it is scaled once more by input gate activation and \(g'\). It then serves to change the incoming weights before it is truncated (see the appendix for formulas).

4.6 计算复杂度。与 Mozer 的聚焦循环反向传播算法（Mozer, 1989）类似，只需存储和更新导数\(∂s_{c_j}/∂w_{il}\)。因此 LSTM 算法非常高效，其更新复杂度为\(O(W)\)，其中 W 是权重数量（细节见附录）。因此，LSTM 和完全循环网络的 BPTT 具有相同的时间步更新复杂度（而 RTRL 的要差得多）。然而，与完整 BPTT 不同，LSTM 在空间和时间上都是局部的：无需在可能无限大的栈中存储序列处理过程中观察到的激活值。

4.6 Computational Complexity. As with Mozer's focused recurrent back-propagation algorithm (Mozer, 1989), only the derivatives \(∂s_{c_j}/∂w_{il}\) need to be stored and updated. Hence the LSTM algorithm is very efficient, with an excellent update complexity of \(O(W)\), where W is the number of weights (see details in the appendix). Hence, LSTM and BPTT for fully recurrent nets have the same update complexity per time step (while RTRL's is much worse). Unlike full BPTT, however, LSTM is local in space and time: there is no need to store activation values observed during sequence processing in a stack with potentially unlimited size.

4.7 滥用问题与解决方案。在学习阶段初期，无需跨时间存储信息也可能降低误差。因此网络倾向于滥用记忆细胞，例如作为偏置细胞（它可能使其激活值恒定，并将传出连接用作其他单元的自适应阈值）。潜在的困难在于，释放被滥用的记忆细胞并使其可用于进一步学习可能需要很长时间。如果两个记忆细胞存储相同（冗余）信息，也会出现类似的“滥用问题”。针对滥用问题至少有两大解决方案：（1）顺序网络构建（例如，Fahlman, 1991）：每当误差停止下降时，向网络添加一个记忆细胞及其对应的门单元（见第 5 节实验 2）；（2）输出门偏置：每个输出门获得负的初始偏置，以将初始记忆细胞激活推向零。偏置更负的记忆细胞会自动“分配”到更晚使用（见第 5 节实验 1、3、4、5 和 6）。

4.7 Abuse Problem and Solutions. In the beginning of the learning phase, error reduction may be possible without storing information over time. The network will thus tend to abuse memory cells, for example, as bias cells (it might make their activations constant and use the outgoing connections as adaptive thresholds for other units). The potential difficulty is that it may take a long time to release abused memory cells and make them available for further learning. A similar “abuse problem” appears if two memory cells store the same (redundant) information. There are at least two solutions to the abuse problem: (1) sequential network construction (e.g., Fahlman, 1991): a memory cell and the corresponding gate units are added to the network whenever the error stops decreasing (see experiment 2 in section 5), and (2) output gate bias: each output gate gets a negative initial bias, to push initial memory cell activations toward zero. Memory cells with more negative bias automatically get “allocated” later (see experiments 1, 3, 4, 5, and 6 in section 5).

4.8 内部状态漂移及其补救措施。如果记忆细胞 cj 的输入大多为正或大多为负，其内部状态 sj 会随时间漂移。这具有潜在危险，因为 \(h'(s_j)\) 会变得极小，导致梯度消失。解决此问题的一种方法是选择适当的函数 h。但例如 \(h(x) = x\) 的缺点在于记忆细胞输出范围无限制。我们解决学习初期漂移问题的简单而有效方法是初始将输入门偏置推向零。尽管 \(h'(s_j)\) 的幅度与 \(y^{in}_j\) 和 \(f'_{in_j}\) 的幅度之间存在权衡，但输入门偏置的潜在负面影响与漂移效应相比可以忽略不计。使用逻辑 sigmoid 激活函数时，似乎无需微调初始偏置，第 5.4 节实验 4 和 5 证实了这一点。

4.8 Internal State Drift and Remedies. If memory cell cj’s inputs are mostly positive or mostly negative, then its internal state sj will tend to drift away over time. This is potentially dangerous, for the \(h'(s_j)\) will then adopt very small values, and the gradient will vanish. One way to circumvent this problem is to choose an appropriate function h. But \(h(x) = x\), for instance, has the disadvantage of unrestricted memory cell output range. Our simple but effective way of solving drift problems at the beginning of learning is initially to bias the input gate toward zero. Although there is a trade-off between the magnitudes of \(h'(s_j)\) on the one hand and of \(y^{in}_j\) and \(f'_{in_j}\) on the other, the potential negative effect of input gate bias is negligible compared to the one of the drifting effect. With logistic sigmoid activation functions, there appears to be no need for fine-tuning the initial bias, as confirmed by experiments 4 and 5 in section 5.4.

另一方面，输入门偏置的潜在负面影响与漂移效应相比可以忽略不计。使用逻辑 sigmoid 激活函数时，似乎无需微调初始偏置，第 5.4 节实验 4 和 5 证实了这一点。

on the other hand, the potential negative effect of input gate bias is negligible compared to the one of the drifting effect. With logistic sigmoid activation functions, there appears to be no need for fine-tuning the initial bias, as confirmed by experiments 4 and 5 in section 5.4.

哪些任务适合证明新型长时间滞后算法的质量？首先，所有训练序列中相关输入信号与对应教师信号之间的最小时间滞后必须很长。事实上，许多以前的循环网络算法有时能从极短的训练序列泛化到极长的测试序列（例如，Pollack, 1991）。但真正的长时间滞后问题在训练集中没有任何短时滞样本。例如，Elman 的训练过程、BPTT、离线 RTRL、在线 RTRL 等方法在真正的长时间滞后问题上都表现极差（如 Hochreiter, 1991; Mozer, 1992）。第二个重要要求是任务必须足够复杂，以至于不能通过简单策略（如随机权重猜测）快速解决。我们最近发现（Schmidhuber & Hochreiter, 1996; Hochreiter & Schmidhuber, 1996, 1997），许多先前工作中使用的长时间滞后任务可以通过简单的随机权重猜测比所提出的算法更快地解决。例如，猜测法解决 Bengio 和 Frasconi 的奇偶问题（1994）的一个变体比 Bengio 等人（1994）以及 Bengio 和 Frasconi（1994）测试的七种方法快得多。对于 Miller 和 Giles 的一些问题（1993）也是如此。当然，这并不意味着猜测法是好算法。它只是意味着一些先前使用的问题并不十分适合证明先前提出算法的质量。我们所有的实验（除实验 1 外）都涉及较长的最小时间滞后；没有短时滞训练样本促进学习。我们大多数任务的解在权重空间中是稀疏的。它们需要大量参数和输入或高权重精度，使得随机权重猜测不可行。我们始终使用在线学习（而非批量学习）和逻辑 sigmoid 作为激活函数。实验 1 和 2 的初始权重选择在 [−0.2, 0.2] 范围内，其他实验在 [−0.1, 0.1] 范围内。训练序列根据各种任务描述随机生成。与附录 A.1 中的符号略有不同，每个输入序列的每个离散时间步涉及三个处理步骤：（1）使用当前输入设置输入单元；（2）计算隐藏单元（包括输入门、输出门、记忆细胞）的激活；（3）计算输出单元激活。除实验 1、2a 和 2b 外，序列元素在线随机生成，且仅在序列末尾生成误差信号。每个输入序列处理后重置网络激活。在与梯度下降训练的循环网络比较时，我们只给出 RTRL 的结果，除非比较 2a 也包含 BPTT。但请注意，未截断的 BPTT（例如，Williams & Peng, 1990）计算与离线 RTRL 完全相同的梯度。对于长时间滞后问题，离线 RTRL（或 BPTT）和在线版本的 RTRL（无激活重置、在线权重更新）产生几乎相同的负面结果（如 Hochreiter, 1991 中的额外模拟所证实；另见 Mozer, 1992）。这是因为离线 RTRL、在线 RTRL 和完整 BPTT 都严重遭受指数级误差衰减。我们的 LSTM 架构选择相当随意。如果对给定问题的复杂度一无所知，更系统的方法是：从一个包含一个记忆细胞的极小网络开始。如果不行，尝试两个细胞，以此类推。或者使用顺序网络构建（例如，Fahlman, 1991）。以下是实验概览：

Which tasks are appropriate to demonstrate the quality of a novel long-time-lag algorithm? First, minimal time lags between relevant input signals and corresponding teacher signals must be long for all training sequences. In fact, many previous recurrent net algorithms sometimes manage to generalize from very short training sequences to very long test sequences (see, e.g., Pollack, 1991). But a real long-time-lag problem does not have any short-time-lag exemplars in the training set. For instance, Elman’s training procedure, BPTT, offline RTRL, online RTRL, and others fail miserably on real long-time-lag problems. (See, e.g., Hochreiter, 1991; Mozer, 1992.) A second important requirement is that the tasks should be complex enough such that they cannot be solved quickly by simple-minded strategies such as random weight guessing. Recently we discovered (Schmidhuber & Hochreiter, 1996; Hochreiter & Schmidhuber, 1996, 1997) that many long-time-lag tasks used in previous work can be solved more quickly by simple random weight guessing than by the proposed algorithms. For instance, guessing solved a variant of Bengio and Frasconi’s parity problem (1994) much faster than the seven methods tested by Bengio et al. (1994) and Bengio and Frasconi (1994). The same is true for some of Miller and Giles’s problems (1993). Of course, this does not mean that guessing is a good algorithm. It just means that some previously used problems are not extremely appropriate to demonstrate the quality of previously proposed algorithms. All our experiments (except experiment 1) involve long minimal time lags; there are no short-time-lag training exemplars facilitating learning. Solutions to most of our tasks are sparse in weight space. They require either many parameters and inputs or high weight precision, such that random weight guessing becomes infeasible. We always use online learning (as opposed to batch learning) and logistic sigmoids as activation functions. For experiments 1 and 2, initial weights are chosen in the range [−0.2, 0.2], for the other experiments in [−0.1, 0.1]. Training sequences are generated randomly according to the various task descriptions. In slight deviation from the notation in appendix A.1, each discrete time step of each input sequence involves three processing steps: (1) use current input to set the input units, (2) compute activations of hidden units (including input gates, output gates, memory cells), and (3) compute output unit activations. Except for experiments 1, 2a, and 2b, sequence elements are randomly generated online, and error signals are generated only at sequence ends. Net activations are reset after each processed input sequence. For comparisons with recurrent nets taught by gradient descent, we give results only for RTRL, except for comparison 2a, which also includes BPTT. Note, however, that untruncated BPTT (see, e.g., Williams & Peng, 1990) computes exactly the same gradient as offline RTRL. With long-time-lag problems, offline RTRL (or BPTT) and the online version of RTRL (no activation resets, online weight changes) lead to almost identical, negative results (as confirmed by additional simulations in Hochreiter, 1991; see also Mozer, 1992). This is because offline RTRL, online RTRL, and full BPTT all suffer badly from exponential error decay. Our LSTM architectures are selected quite arbitrarily. If nothing is known about the complexity of a given problem, a more systematic approach would be to: start with a very small net consisting of one memory cell. If this does not work, try two cells, and so on. Alternatively, use sequential network construction (e.g., Fahlman, 1991). Following is an outline of the experiments:

• 实验 1 关注循环网络的一个标准基准测试：嵌入式 Reber 语法。由于它允许训练序列具有短时间滞后，因此不是长时间滞后问题。我们包含它是因为它提供了一个很好的例子，展示了 LSTM 的输出门真正有用，而且它是许多作者使用的流行循环网络基准。我们希望至少包含一个传统 BPTT 和 RTRL 不会完全失败的实验（但 LSTM 明显优于它们）。嵌入式 Reber 语法的最小时滞代表一种边界情况，即传统算法仍可能学习跨越它们。只要最小时滞稍长，这几乎就不可能了。然而，我们文章中更有趣的任务是那些 RTRL、BPTT 等方法根本无法解决的任务。

• Experiment 1 focuses on a standard benchmark test for recurrent nets: the embedded Reber grammar. Since it allows for training sequences with short time lags, it is not a long-time-lag problem. We include it because it provides a nice example where LSTM’s output gates are truly beneficial, and it is a popular benchmark for recurrent nets that has been used by many authors. We want to include at least one experiment where conventional BPTT and RTRL do not fail completely (LSTM, however, clearly outperforms them). The embedded Reber grammar’s minimal time lags represent a border case in the sense that it is still possible to learn to bridge them with conventional algorithms. Only slightly longer minimal time lags would make this almost impossible. The more interesting tasks in our article, however, are those that RTRL, BPTT, and others cannot solve at all.

• 实验 2 关注无噪声和含噪声序列，其中包含大量输入符号，干扰着少数重要符号。最困难的任务（任务 2c）涉及数百个随机位置的干扰符号以及最小时间滞后 1000 步。LSTM 解决了该任务；而 BPTT 和 RTRL 在最小时间滞后仅为 10 步时就已经失败（另见 Hochreiter, 1991; Mozer, 1992）。因此，在剩余更复杂的实验中（所有实验都涉及更长的时间滞后），RTRL 和 BPTT 被省略。1750 Sepp Hochreiter and Jürgen Schmidhuber

• Experiment 2 focuses on noise-free and noisy sequences involving numerous input symbols distracting from the few important ones. The most difficult task (task 2c) involves hundreds of distractor symbols at random positions and minimal time lags of 1000 steps. LSTM solves it; BPTT and RTRL already fail in case of 10-step minimal time lags (see also Hochreiter, 1991; Mozer, 1992). For this reason RTRL and BPTT are omitted in the remaining, more complex experiments, all of which involve much longer time lags. 1750 Sepp Hochreiter and Jürgen Schmidhuber

• 实验 3 处理同一输入线上同时存在噪声和信号的长时滞问题。实验 3a 和 3b 聚焦于 Bengio 等人 1994 年的双序列问题。由于该问题可以通过随机权重猜测快速解决，我们还包含了一个更困难的双序列问题（实验 3c），它要求根据输入学习含噪目标的实值条件期望。

• Experiment 3 addresses long-time-lag problems with noise and signal on the same input line. Experiments 3a and 3b focus on Bengio et al.’s 1994 two-sequence problem. Because this problem can be solved quickly by random weight guessing, we also include a far more difficult two-sequence problem (experiment 3c), which requires learning real-valued, conditional expectations of noisy targets, given the inputs.

• 实验 4 和 5 涉及分布式、连续值的输入表示，并要求学习在极长时间段内存储精确的实数值。相关的输入信号可能出现在输入序列中截然不同的位置。同样，最小时间滞后涉及数百步。类似的任务从未被其他循环网络算法解决过。

• Experiments 4 and 5 involve distributed, continuous-valued input representations and require learning to store precise, real values for very long time periods. Relevant input signals can occur at quite different positions in input sequences. Again minimal time lags involve hundreds of steps. Similar tasks never have been solved by other recurrent net algorithms.

• 实验 6 涉及另一类复杂任务，也未被其他循环网络算法解决。同样，相关输入信号可能出现在输入序列中截然不同的位置。实验表明，LSTM 能够提取由广泛分离的输入的时间顺序所传达的信息。第 5.7 节以两个表格的形式提供了实验条件的详细总结，以供参考。

• Experiment 6 involves tasks of a different complex type that also has not been solved by other recurrent net algorithms. Again, relevant input signals can occur at quite different positions in input sequences. The experiment shows that LSTM can extract information conveyed by the temporal order of widely separated inputs. Section 5.7 provides a detailed summary of experimental conditions in two tables for reference.

5.1 实验 1：嵌入式 Reber 文法

5.1 Experiment 1: Embedded Reber Grammar.

5.1.1 任务。我们的第一个任务是学习嵌入式 Reber 语法（Smith & Zipser, 1989; Cleeremans, Servan-Schreiber, & McClelland, 1989; Fahlman, 1991）。由于该任务允许训练短时滞序列（最少仅九步），因此并非长时间滞问题。我们纳入该任务有两个原因：（1）它是一个被许多作者使用的流行循环网络基准，我们希望至少有一个实验，其中 RTRL 和 BPTT 不会完全失败；（2）它能很好地展示输出门如何发挥作用。从图 3 有向图的最左边节点开始，通过沿着边移动并将关联符号附加到当前字符串上，顺序生成符号字符串（从空字符串开始），直到到达最右边节点（Reber 语法子串类似地从图 4 生成）。若存在多条边，则随机选择（概率为 0.5）。网络的任务是逐个读取符号并预测下一个符号（每个时间步都会产生误差信号）。要预测倒数第二个符号，网络必须记住第二个符号。

5.1.1 Task. Our first task is to learn the embedded Reber grammar (Smith & Zipser, 1989; Cleeremans, Servan-Schreiber, & McClelland, 1989; Fahlman, 1991). Since it allows for training sequences with short time lags (of as few as nine steps), it is not a long-time-lag problem. We include it for two reasons: (1) it is a popular recurrent net benchmark used by many authors, and we wanted to have at least one experiment where RTRL and BPTT do not fail completely, and (2) it shows nicely how output gates can be beneficial. Starting at the left-most node of the directed graph in Figure 3, symbol strings are generated sequentially (beginning with the empty string) by following edges—and appending the associated symbols to the current string—until the right-most node is reached (the Reber grammar substrings are analogously generated from Figure 4). Edges are chosen randomly if there is a choice (probability: 0.5). The net’s task is to read strings, one symbol at a time, and to predict the next symbol (error signals occur at every time step). To predict the symbol before last, the net has to remember the second symbol.

5.1.2 比较。我们将 LSTM 与通过 Elman 训练程序（ELM）训练的 Elman 网络（结果取自 Cleeremans 等，1989）、Fahlman 的循环级联相关（RCC）（结果取自 Fahlman，1991）进行比较。

5.1.2 Comparison. We compare LSTM to Elman nets trained by Elman’s training procedure (ELM) (results taken from Cleeremans et al., 1989), Fahlman’s recurrent cascade-correlation (RCC) (results taken from Fahlman, 1991).

Reber 语法转换图 BTS XX PVTP VSE

图 3：Reber 语法的转换图。

Figure 3: Transition diagram for the Reber grammar.

嵌入式 Reber 语法 BTPETPGRAMMAR GRAMMAR REBER REBER

图 4：嵌入式 Reber 语法的转移图。每个方框代表 Reber 语法的一个副本（见图 3）。Sepp Hochreiter 和 Jürgen Schmidhuber (1991)，以及 RTRL（结果取自 Smith & Zipser, 1989），其中只列出了少数成功的试验。Smith 和 Zipser 实际上通过增加短时滞样本的概率使任务变得更容易。我们在 LSTM 中没有这样做。

Figure 4: Transition diagram for the embedded Reber grammar. Each box represents a copy of the Reber grammar (see Figure 3). Sepp Hochreiter and Jürgen Schmidhuber (1991), and RTRL (results taken from Smith & Zipser, 1989), where only the few successful trials are listed. Smith and Zipser actually make the task easier by increasing the probability of short-time-lag exemplars. We did not do this for LSTM.

5.1.3 训练/测试。我们使用局部输入-输出表示（七个输入单元，七个输出单元）。遵循 Fahlman 的做法，我们使用 256 个训练字符串和 256 个独立的测试字符串。训练集是随机生成的；训练样本从训练集中随机选取。测试序列也是随机生成的，但训练集中已经使用过的序列不会用于测试。字符串呈现后，所有激活值都重新初始化为零。如果测试集和训练集中所有序列的所有字符串符号都被正确预测——即对应于可能的下一个符号的输出单元总是最活跃的——则认为一次试验成功。

5.1.3 Training/Testing. We use a local input-output representation (seven input units, seven output units). Following Fahlman, we use 256 training strings and 256 separate test strings. The training set is generated randomly; training exemplars are picked randomly from the training set. Test sequences are generated randomly, too, but sequences already used in the training set are not used for testing. After string presentation, all activations are reinitialized with zeros. A trial is considered successful if all string symbols of all sequences in both test set and training set are predicted correctly—that is, if the output unit(s) corresponding to the possible next symbol(s) are always the most active ones.

5.1.4 架构。RTRL、ELM 和 RCC 的架构见上文引用的文献。对于 LSTM，我们使用三个（四个）记忆单元块。每个块有两个（一个）记忆单元。输出层的唯一传入连接来自记忆单元。每个记忆单元和每个门单元接收来自所有记忆单元和门单元的传入连接（隐藏层是全连接的；较少连接也可能有效）。输入层向前连接到隐藏层中的所有单元。门单元有偏置。这些架构参数使得很容易存储至少三个输入信号（采用架构 3-2 和 4-1 以获得两个架构可比较的权重数量：4-1 为 264，3-2 为 276）。然而，其他参数也可能合适。所有 sigmoid 函数都是 logistic 函数，输出范围为[0, 1]，除了 h 的范围为[−1, 1]，g 的范围为[−2, 2]。所有权重初始化为[−0.2, 0.2]，除了输出门偏置分别初始化为−1、−2 和−3（参见滥用问题，第 4 节的解决方案 2）。我们尝试了学习率 0.1、0.2 和 0.5。

5.1.4 Architectures. Architectures for RTRL, ELM, and RCC are reported in the references listed above. For LSTM, we use three (four) memory cell blocks. Each block has two (one) memory cells. The output layer’s only incoming connections originate at memory cells. Each memory cell and each gate unit receives incoming connections from all memory cells and gate units (the hidden layer is fully connected; less connectivity may work as well). The input layer has forward connections to all units in the hidden layer. The gate units are biased. These architecture parameters make it easy to store at least three input signals (architectures 3-2 and 4-1 are employed to obtain comparable numbers of weights for both architectures: 264 for 4-1 and 276 for 3-2). Other parameters may be appropriate as well, however. All sigmoid functions are logistic with output range [0, 1], except for h, whose range is [−1, 1], and g, whose range is [−2, 2]. All weights are initialized in [−0.2, 0.2], except for the output gate biases, which are initialized to −1, −2, and −3, respectively (see abuse problem, solution 2 of section 4). We tried learning rates of 0.1, 0.2, and 0.5.

5.1.5 结果。我们使用三对不同随机生成的训练集和测试集。对每一对，我们使用不同的初始权重运行 10 次试验。结果见表 1（30 次试验的平均值）。与其他方法不同，LSTM 总能学会解决任务。即使忽略其他方法不成功的试验，LSTM 学习速度也快得多。

5.1.5 Results. We use three different, randomly generated pairs of training and test sets. With each such pair we run 10 trials with different initial weights. See Table 1 for results (mean of 30 trials). Unlike the other methods, LSTM always learns to solve the task. Even when we ignore the unsuccessful trials of the other approaches, LSTM learns much faster.

5.1.6 输出门的重要性。该实验很好地说明了输出门确实是有益的。学习存储第一个 T 或 P 不应扰动代表原始 Reber 语法中更易学习的转移的激活值。这就是输出门的工作。没有输出门，我们无法实现快速学习。表 1：实验 1：嵌入式 Reber 语法。学习方法 隐藏单元数 权重数 学习率 成功率% 成功所需呈现次数 RTRL 3 ≈170 0.05 部分 173,000 RTRL 12 ≈494 0.1 部分 25,000 ELM 15 ≈435 0 >200,000 RCC 7–9 ≈119–198 50 182,000 LSTM 4 块，大小 1 264 0.1 100 39,740 LSTM 3 块，大小 2 276 0.1 100 21,730 LSTM 3 块，大小 2 276 0.2 97 14,060 LSTM 4 块，大小 1 264 0.5 97 9,500 LSTM 3 块，大小 2 276 0.5 100 8,440 注：成功试验的百分比和成功所需的序列呈现次数，RTRL（结果取自 Smith & Zipser, 1989），Elman 网络通过 Elman 过程训练（结果取自 Cleeremans 等，1989），递归级联相关（结果取自 Fahlman, 1991），以及我们的新方法（LSTM）。前四行的权重数是估计值，相应论文未提供所有技术细节。只有 LSTM 几乎总能学会解决任务（150 次试验中仅 2 次失败）。即使忽略其他方法不成功的试验，LSTM 学习速度也快得多（最后一行所需训练样本数在 3,800 到 24,100 之间）。

5.1.6 Importance of Output Gates. The experiment provides a nice example where the output gate is truly beneficial. Learning to store the first T or P should not perturb activations representing the more easily learnable transitions of the original Reber grammar. This is the job of the output gates. Without output gates, we did not achieve fast learning. Table 1: Experiment 1: Embedded Reber Grammar. Number of Learning Method Hidden Units Weights Rate % of Success After RTRL 3 ≈ 170 0.05 Some fraction 173,000 RTRL 12 ≈ 494 0.1 Some fraction 25,000 ELM 15 ≈ 435 0 >200,000 RCC 7–9 ≈ 119–198 50 182,000 LSTM 4 blocks, size 1 264 0.1 100 39,740 LSTM 3 blocks, size 2 276 0.1 100 21,730 LSTM 3 blocks, size 2 276 0.2 97 14,060 LSTM 4 blocks, size 1 264 0.5 97 9500 LSTM 3 blocks, size 2 276 0.5 100 8440 Notes: Percentage of successful trials and number of sequence presentations until success for RTRL (results taken from Smith & Zipser, 1989), Elman net trained by Elman’s procedure (results taken from Cleeremans et al., 1989), recurrent cascade-correlation (results taken from Fahlman, 1991), and our new approach (LSTM). Weight numbers in the first four rows are estimates, the corresponding papers do not provide all the technical details. Only LSTM almost always learns to solve the task (only 2 failures out of 150 trials). Even when we ignore the unsuccessful trials of the other approaches, LSTM learns much faster (the number of required training examples in the bottom row varies between 3800 and 24,100).

5.2.1 任务 2a：无噪声长时滞序列。共有 p+1 个可能的输入符号，记为 a₁, …, a_{p-1}, a_p = x, a_{p+1} = y。每个 a_i 由 p+1 维向量局部表示，其中第 i 个分量为 1（其他分量为 0）。网络具有 p+1 个输入单元和 p+1 个输出单元，依次逐个观察输入符号序列，并持续预测下一个符号；每个时间步都产生误差信号。为突出长时滞问题，我们使用仅包含两个非常相似序列的训练集：(y, a₁, a₂, …, a_{p-1}, y)和(x, a₁, a₂, …, a_{p-1}, x)，每个序列以 0.5 的概率被选择。为预测最后一个元素，网络必须学习在 p 个时间步内存储第一个元素的表示。我们比较了全递归网络的实时递归学习（RTRL）、随时间反向传播（BPTT）、有时非常成功的双网络神经序列分块器（CH；Schmidhuber, 1992b）以及我们的新方法（LSTM）。所有情况下，权重初始化为\[-0.2, 0.2\]。由于计算时间有限，训练在 500 万次序列呈现后停止。成功的运行满足以下标准：训练后，在 10,000 个连续随机选择的输入序列中，所有输出单元的最大绝对误差始终低于 0.25。1754 Sepp Hochreiter 和 Jürgen Schmidhuber 表 2：任务 2a：成功试验百分比和成功所需训练序列数。学习方法 延迟 p 学习率 权重数 成功试验百分比 成功所需序列数 RTRL 4 1.0 36 78 1,043,000 RTRL 4 4.0 36 56 892,000 RTRL 4 10.0 36 22 254,000 RTRL 10 1.0–10.0 144 0 >5,000,000 RTRL 100 1.0–10.0 10,404 0 >5,000,000 BPTT 100 1.0–10.0 10,404 0 >5,000,000 CH 100 1.0 10,506 33 32,400 LSTM 100 1.0 10,504 100 5,040 注：表格条目为 18 次试验的平均值。在 100 个时间步延迟下，只有 CH 和 LSTM 取得了成功试验。即使忽略其他方法的不成功试验，LSTM 也学习得快得多。

5.2.1 Task 2a: Noise-Free Sequences with Long Time Lags. There are p+1 possible input symbols denoted a1, ..., a_{p-1}, a_p = x, a_{p+1} = y. a_i is locally represented by the p+1-dimensional vector whose ith component is 1 (all other components are 0). A net with p+1 input units and p+1 output units sequentially observes input symbol sequences, one at a time, permanently trying to predict the next symbol; error signals occur at every time step. To emphasize the long-time-lag problem, we use a training set consisting of only two very similar sequences: (y, a_1, a_2, ..., a_{p-1}, y) and (x, a_1, a_2, ..., a_{p-1}, x). Each is selected with probability 0.5. To predict the final element, the net has to learn to store a representation of the first element for p time steps. We compare real-time recurrent learning for fully recurrent nets (RTRL), back-propagation through time (BPTT), the sometimes very successful two-net neural sequence chunker (CH; Schmidhuber, 1992b), and our new method (LSTM). In all cases, weights are initialized in \[-0.2, 0.2\]. Due to limited computation time, training is stopped after 5 million sequence presentations. A successful run is one that fulfills the following criterion: after training, during 10,000 successive, randomly chosen input sequences, the maximal absolute error of all output units is always below 0.25. 1754 Sepp Hochreiter and Jürgen Schmidhuber Table 2: Task 2a: Percentage of Successful Trials and Number of Training Sequences until Success. Learning Number of % Successful Success Method Delay p Rate Weights Trials After RTRL 4 1.0 36 78 1,043,000 RTRL 4 4.0 36 56 892,000 RTRL 4 10.0 36 22 254,000 RTRL 10 1.0–10.0 144 0 > 5,000,000 RTRL 100 1.0–10.0 10404 0 > 5,000,000 BPTT 100 1.0–10.0 10404 0 > 5,000,000 CH 100 1.0 10506 33 32,400 LSTM 100 1.0 10504 100 5,040 Notes: Table entries refer to means of 18 trials. With 100 time-step delays, only CH and LSTM achieve successful trials. Even when we ignore the unsuccessful trials of the other approaches, LSTM learns much faster.

RTRL：一个自递归隐藏单元，p+1 个非递归输出单元。每一层与所有下层有连接。所有单元使用逻辑激活函数 sigmoid，输出在\[0, 1\]内。BPTT：与 RTRL 训练的架构相同。CH：两种网络架构与 RTRL 相同，但其中一个网络有一个额外的输出来预测另一个网络的隐藏单元（详见 Schmidhuber, 1992b）。LSTM：与 RTRL 类似，但隐藏单元被替换为记忆单元和输入门（无需输出门）。g 是逻辑 sigmoid 函数，h 是恒等函数\(h(x)=x, \forall x\)。当误差停止下降时，添加记忆单元和输入门（参见第 4 节中的滥用问题：解决方案 1）。

RTRL: One self-recurrent hidden unit, p+1 nonrecurrent output units. Each layer has connections from all layers below. All units use the logistic activation function sigmoid in \[0, 1\]. BPTT: Same architecture as the one trained by RTRL. CH: Both net architectures like RTRL's, but one has an additional output for predicting the hidden unit of the other one (see Schmidhuber, 1992b, for details). LSTM: As with RTRL, but the hidden unit is replaced by a memory cell and an input gate (no output gate required). g is the logistic sigmoid, and h is the identity function \(h(x)=x, \forall x\). Memory cell and input gate are added once the error has stopped decreasing (see abuse problem: solution 1 in section 4).

结果。使用 RTRL 和短的四时间步延迟（p=4），7/9 的试验成功。p=10 时没有试验成功。对于长时滞，只有神经序列分块器和 LSTM 取得了成功试验；BPTT 和 RTRL 失败。当 p=100 时，双网络序列分块器仅在三分之一的试验中解决了任务。然而，LSTM 始终能学会解决任务。仅比较成功试验时，LSTM 学习速度更快。详见 Table 2。但需指出，层次化分块器也能始终快速解决此任务（Schmidhuber, 1992c, 1993）。

Results. Using RTRL and a short four-time-step delay (p=4), 7/9 of all trials were successful. No trial was successful with p=10. With long time lags, only the neural sequence chunker and LSTM achieved successful trials; BPTT and RTRL failed. With p=100, the two-net sequence chunker solved the task in only one-third of all trials. LSTM, however, always learned to solve the task. Comparing successful trials only, LSTM learned much faster. See Table 2 for details. It should be mentioned, however, that a hierarchical chunker can also always quickly solve this task (Schmidhuber, 1992c, 1993).

5.2 Experiment 2: Noise-Free and Noisy Sequences.

5.2.1 Task 2a: Noise-Free Sequences with Long Time Lags. There are p + 1possible input symbols denoted a1, . . . , ap−1, ap = x, ap+1 = y. ai is locally represented by the p + 1-dimensional vector whose ith component is 1 (all other components are 0). A net with p + 1 input units and p + 1 output units sequentially observes input symbol sequences, one at a time, per-manently trying to predict the next symbol; error signals occur at every time step. To emphasize the long-time-lag problem, we use a training set consisting of only two very similar sequences: (y, a1, a2, . . . , ap−1, y) and

5.2.2 任务 2b：无局部规律性。在任务 2a 中，分块器有时能够正确预测最终元素，但这仅仅是因为输入流中存在 pre-Long Short-Term Memory 1755 可预测的局部规律性，从而允许压缩序列。在一个更困难的任务中，涉及更多不同的可能序列，我们通过将确定性子序列\(a_1, a_2, \ldots, a_{p-1}\)替换为随机子序列（长度为

5.2.2 Task 2b: No Local Regularities. With task 2a, the chunker sometimes learns to predict the final element correctly, but only because of pre-Long Short-Term Memory 1755 predictable local regularities in the input stream that allow for compressing the sequence. In a more difficult task, involving many more different possible sequences, we remove compressibility by replacing the deterministic subsequence \(a_1, a_2, \ldots, a_{p-1}\) by a random subsequence (of length

p−1），字母表为\(a_1, a_2, \ldots, a_{p-1}\)。我们得到两类（两个序列集合）\{(y, a_{i_1}, a_{i_2}, \ldots, a_{i_{p-1}}, y) \mid 1 \leq i_1, i_2, \ldots, i_{p-1} \leq p-1}\ 和

p - 1) over the alphabet \(a_1, a_2, \ldots, a_{p-1}\). We obtain two classes (two sets of sequences) \{(y, a_{i_1}, a_{i_2}, \ldots, a_{i_{p-1}}, y) \mid 1 \leq i_1, i_2, \ldots, i_{p-1} \leq p-1}\ and

\{(x, a_{i_1}, a_{i_2}, \ldots, a_{i_{p-1}}, x) \mid 1 \leq i_1, i_2, \ldots, i_{p-1} \leq p-1}\。同样，每个后续序列元素都必须被预测。然而，唯一完全可预测的目标是 x 和 y，它们出现在序列末尾。训练样本从这两类中随机选择。架构和参数与实验 2a 相同。成功的运行需满足以下标准：训练后，在连续 10000 个随机选择的输入序列中，所有输出单元在序列末尾的最大绝对误差低于 0.25。

\{(x, a_{i_1}, a_{i_2}, \ldots, a_{i_{p-1}}, x) \mid 1 \leq i_1, i_2, \ldots, i_{p-1} \leq p-1}\. Again, every next sequence element has to be predicted. The only totally predictable targets, however, are x and y, which occur at sequence ends. Training exemplars are chosen randomly from the two classes. Architectures and parameters are the same as in experiment 2a. A successful run is one that fulfills the following criterion: after training, during 10,000 successive, randomly chosen input sequences, the maximal absolute error of all output units is below 0.25 at sequence end.

结果。正如预期，分块器无法解决此任务（BPTT 和 RTRL 当然也失败）。然而，LSTM 总是成功。平均而言（18 次试验的均值），对于 p=100，在 5680 次序列呈现后达到成功。这表明 LSTM 不需要序列规律性也能很好地工作。

Results. As expected, the chunker failed to solve this task (so did BPTT and RTRL, of course). LSTM, however, was always successful. On average (mean of 18 trials), success for p = 100 was achieved after 5680 sequence presentations. This demonstrates that LSTM does not require sequence regularities to work well.

5.2.3 任务 2c：极长时滞——无局部规律性。这是本小节中最困难的任务。据我们所知，没有其他循环网络算法能够解决它。现在有 p+4 个可能的输入符号，记为\(a_1, \ldots, a_{p-1}, a_p, a_{p+1}=e, a_{p+2}=b, a_{p+3}=x, a_{p+4}=y\)。\(a_1, \ldots, a_p\

5.2.3 Task 2c: Very Long Time Lags—No Local Regularities. This is the most difficult task in this subsection. To our knowledge, no other recurrent net algorithm can solve it. Now there are p + 4 possible input symbols denoted \(a_1, \ldots, a_{p-1}, a_p, a_{p+1}=e, a_{p+2}=b, a_{p+3}=x, a_{p+4}=y\). \(a_1, \ldots, a_p\

也称为干扰符号。同样，a_i 由 p+4 维向量局部表示，其第 i 分量为 1（所有其他分量为 0）。

are also called distractor symbols. Again, ai is locally represented by the p + 4-dimensional vector whose ith component is 1 (all other components are 0).

一个具有 p+4 个输入单元和 2 个输出单元的网络逐个顺序观察输入符号序列。训练序列从两个非常相似的序列子集的并集中随机选择：

A net with p + 4 input units and 2 output units sequentially observes input symbol sequences, one at a time. Training sequences are randomly chosen from the union of two very similar subsets of sequences:

{(b, y, a_{i1}, a_{i2}, ..., a_{i_{q+k}}, e, y) | 1 ≤ i1, i2, ..., i_{q+k} ≤ q} 和 {(b, x, a_{i1}, a_{i2}, ..., a_{i_{q+k}}, e, x) | 1 ≤ i1, i2, ..., i_{q+k} ≤ q}。

{(b, y, ai1 , ai2 , . . . , a iq+k , e, y) | 1 ≤ i1, i2, . . . , i q+k ≤ q} and {(b, x, ai1 , ai2 , . . . , a i q+k , e, x) | 1 ≤ i1, i2, . . . , i q+k ≤ q}.

为了生成训练序列，我们随机生成一个长度为 q+2 的序列前缀，然后以 9/10 的概率随机生成一个由额外元素（≠ b, e, x, y）组成的后缀，或者以 1/10 的概率生成一个 e。在后一种情况下，序列以 x 或 y 结尾，具体取决于第二个元素。对于给定的 k，这导致长度为 q+k+4 的可能序列服从均匀分布。

To produce a training sequence, we randomly generate a sequence prefix of length q + 2, randomly generate a sequence suffix of additional elements (≠ b, e, x, y) with probability 9/10 or, alternatively, an e with probability 1/10. In the latter case, we conclude the sequence with x or y, depending on the second element. For a given k, this leads to a uniform distribution on the possible sequences with length q + k + 4.

最小序列长度为 q+4；期望长度为 4 + \sum_{i=1}^{\infty} ...

The minimal sequence length is q + 4; the expected length is 4 + \sum_{i=1}^{\infty} ...

序列中元素 \a_i\（\1 \le i \le p\）的期望出现次数为 \(q+10)/p \approx q/p\。目标是预测最后一个符号，该符号总是在“触发符号”e 之后出现。误差信号仅在序列结束时生成。表 3：任务 2c：具有极长最小时间滞后 q+1 和大量噪声的 LSTM。

The expected number of occurrences of element \a_i\, \1 \le i \le p\, in a sequence is \(q+10)/p \approx q/p\. The goal is to predict the last symbol, which always occurs after the “trigger symbol” e. Error signals are generated only at sequence ends. Table 3: Task 2c: LSTM with Very Long Minimal Time Lags q+1 and a Lot of Noise.

为了预测最终元素，网络必须学会存储第二个元素的表示至少 q+1 个时间步（直到它看到触发符号 e）。成功定义为两个输出单元对最终序列元素的预测误差始终低于 0.2，持续 10,000 个连续随机选择的输入序列。

To predict the final element, the net has to learn to store a representation of the second element for at least q+1 time steps (until it sees the trigger symbol e). Success is defined as prediction error (for final sequence element) of both output units always below 0.2, for 10,000 successive, randomly chosen input sequences.

架构/学习。网络有 p+4 个输入单元和 2 个输出单元。权重初始化为[-0.2, 0.2]。为了避免因不同权重初始化导致的学习时间方差过大，隐藏层有两个记忆细胞（两个大小为 1 的细胞块，尽管一个就足够了）。没有其他隐藏单元。输出层仅从记忆细胞接收连接。记忆细胞和门单元从输入单元、记忆细胞和门单元接收连接（隐藏层全连接）。不使用偏置权重。h 和 g 是逻辑 Sigmoid 函数，输出范围分别为[-1, 1]和[-2, 2]。学习率为 0.01。注意最小时间滞后为 q+1；网络从未见过有助于长测试序列分类的短训练序列。

Architecture/Learning. The net has p+4 input units and 2 output units. Weights are initialized in [-0.2, 0.2]. To avoid too much learning time variance due to different weight initializations, the hidden layer gets two memory cells (two cell blocks of size 1, although one would be sufficient). There are no other hidden units. The output layer receives connections only from memory cells. Memory cells and gate units receive connections from input units, memory cells, and gate units (the hidden layer is fully connected). No bias weights are used. h and g are logistic sigmoids with output ranges [-1, 1] and [-2, 2], respectively. The learning rate is 0.01. Note that the minimal time lag is q+1; the net never sees short training sequences facilitating the classification of long test sequences.

结果。对所有测试对(p, q)进行了二十次试验。表 3 列出了 LSTM 成功所需的训练序列数量的平均值（BPTT 和 RTRL 没有机会解决最小时间滞后为 1000 步的非平凡任务）。

Results. Twenty trials were made for all tested pairs (p, q). Table 3 lists the mean of the number of training sequences required by LSTM to achieve success (BPTT and RTRL have no chance of solving nontrivial tasks with minimal time lags of 1000 steps).

规模扩展。表 3 显示，如果我们让输入符号（和权重）的数量与时间滞后成比例增加，学习时间增长得非常缓慢。这是 LSTM 的另一个显著特性，我们知道的任何其他方法都不具备。事实上，RTRL 和 BPTT 远远不能合理地扩展；相反，它们似乎呈指数级扩展，并且当时间滞后仅超过 10 步时就显得相当无用。

Scaling. Table 3 shows that if we let the number of input symbols (and weights) increase in proportion to the time lag, learning time increases very slowly. This is another remarkable property of LSTM not shared by any other method we are aware of. Indeed, RTRL and BPTT are far from scaling reasonably; instead, they appear to scale exponentially and appear quite useless when the time lags exceed as few as 10 steps.

干扰物影响。在表 3 中，以 q/p 为表头的列给出了干扰符号的预期频率。增加这一频率会降低学习速度，这是由经常观察到的输入符号引起的权重振荡所致。

Distractor influence. In Table 3, the column headed by q/p gives the expected frequency of distractor symbols. Increasing this frequency decreases learning speed, an effect due to weight oscillations caused by frequently observed input symbols.

5.3 实验 3：同一通道上的噪声与信号。本实验旨在说明，如果噪声和信号混合在同一输入线上，LSTM 不会遇到根本性问题。我们首先关注 Bengio 等人（1994 年）的简单两序列问题。在实验 3c 中，我们提出了一个更具挑战性的两序列问题。

5.3 Experiment 3: Noise and Signal on Same Channel. This experiment serves to illustrate that LSTM does not encounter fundamental problems if noise and signal are mixed on the same input line. We initially focus on Bengio et al.'s simple 1994 two-sequence problem. In experiment 3c we pose a more challenging two-sequence problem.

5.3.1 任务 3a（两序列问题）。该任务要求观察并对输入序列进行分类。共有两个类别，每个类别出现的概率为 0.5。只有一条输入线。只有前 N 个实值序列元素携带关于类别的相关信息。位置 t > N 的序列元素由均值为 0、方差为 0.2 的高斯分布生成。N=1 的情况：第一个序列元素对于类别 1 为 1.0，对于类别 2 为-1.0。N=3 的情况：前三个元素对于类别 1 为 1.0，对于类别 2 为-1.0。序列末尾的目标对于类别 1 为 1.0，对于类别 2 为 0.0。正确分类定义为序列末尾的绝对输出误差低于 0.2。给定常数 T，序列长度在 T 到 T+T/10 之间随机选择（与 Bengio 等人问题的一个区别是，他们也允许长度为 T/2 的更短序列）。

5.3.1 Task 3a (Two-Sequence Problem). The task is to observe and then classify input sequences. There are two classes, each occurring with probability 0.5. There is only one input line. Only the first N real-valued sequence elements convey relevant information about the class. Sequence elements at positions t > N are generated by a Gaussian with mean zero and variance 0.2. Case N = 1: the first sequence element is 1.0 for class 1, and -1.0 for class 2. Case N = 3: the first three elements are 1.0 for class 1 and -1.0 for class 2. The target at the sequence end is 1.0 for class 1 and 0.0 for class 2. Correct classification is defined as absolute output error at sequence end below 0.2. Given a constant T, the sequence length is randomly selected between T and T + T/10 (a difference to Bengio et al.'s problem is that they also permit shorter sequences of length T/2).

猜测。Bengio 等人（1994 年）以及 Bengio 和 Frasconi（1994 年）在两序列问题上测试了七种不同的方法。然而，我们发现随机权重猜测很容易胜过所有这些方法，因为该问题过于简单。5 更多类似结果可参见 Schmidhuber 和 Hochreiter（1996 年）以及 Hochreiter 和 Schmidhuber（1996 年, 1997 年）。

Guessing. Bengio et al. (1994) and Bengio and Frasconi (1994) tested seven different methods on the two-sequence problem. We discovered, however, that random weight guessing easily outperforms them all because the problem is so simple. 5 See Schmidhuber and Hochreiter (1996) and Hochreiter and Schmidhuber (1996, 1997) for additional results in this vein.

LSTM 架构。我们使用一个三层网络，包括一个输入单元、一个输出单元和三个大小为 1 的细胞块。输出层仅从记忆细胞接收连接。记忆细胞和门单元接收来自输入单元、记忆细胞和门单元的输入，并具有偏置权重。Gate 5 然而，不同的输入表示和不同类型的噪声可能导致更差的猜测性能（Yoshua Bengio，个人通信，1996 年）。1758 Sepp Hochreiter 和 Jürgen Schmidhuber 表 4：任务 3a：Bengio 等人的两序列问题。编号 ST2：分数

LSTM Architecture. We use a three-layer net with one input unit, one output unit, and three cell blocks of size 1. The output layer receives connections only from memory cells. Memory cells and gate units receive inputs from input units, memory cells, and gate units and have bias weights. Gate 5 However, different input representations and different types of noise may lead to worse guessing performance (Yoshua Bengio, personal communication, 1996). 1758 Sepp Hochreiter and Jürgen Schmidhuber Table 4: Task 3a: Bengio et al.'s Two-Sequence Problem. Number ST2: Fraction

单元和输出单元的激活函数为\[0,1]\上的 logistic sigmoid，h 在\[−1,1]\，g 在\[−2,2]\。

Units and output unit are logistic sigmoid in \[0, 1]\, h in \[−1, 1]\, and g in \[−2, 2]\.

训练/测试。所有权重（除门单元的偏置权重外）在范围\[−0.1,0.1]\内随机初始化。第一个输入门偏置初始化为−1.0，第二个为−3.0，第三个为−5.0。第一个输出门偏置初始化为−2.0，第二个为−4.0，第三个为−6.0。不过，正如额外实验所证实的，精确的初始化值几乎无关紧要。学习率为 1.0。在新序列开始时，所有激活值重置为零。我们根据以下标准停止训练（并判断任务是否解决）：ST1：来自随机选择测试集的 256 个序列均未误分类；ST2：满足 ST1，且测试集平均绝对误差低于 0.01。在 ST2 情况下，使用由 2560 个随机选择序列组成的额外测试集来确定误分类序列的比例。

Training/Testing. All weights (except the bias weights to gate units) are randomly initialized in the range \[−0.1, 0.1]\. The first input gate bias is initialized with −1.0, the second with −3.0, and the third with −5.0. The first output gate bias is initialized with −2.0, the second with −4.0, and the third with −6.0. The precise initialization values hardly matter though, as confirmed by additional experiments. The learning rate is 1.0. All activations are reset to zero at the beginning of a new sequence. We stop training (and judge the task as being solved) according to the following criteria: ST1: none of 256 sequences from a randomly chosen test set is misclassified; ST2: ST1 is satisfied, and mean absolute test set error is below 0.01. In case of ST2, an additional test set consisting of 2560 randomly chosen sequences is used to determine the fraction of misclassified sequences.

结果。见表 4。结果是 10 次试验的平均值，权重初始化为范围\[−0.1,0.1]\内的不同值。LSTM 能够解决这个问题，但远不如随机权重猜测快（见上文“猜测”）。显然，这个平凡问题并不能提供一个很好的测试平台来比较各种非平凡算法的性能。尽管如此，它证明了 LSTM 在面对同一信道上的信号和噪声时不会遇到根本性问题。

Results. See Table 4. The results are means of 10 trials with different weight initializations in the range \[−0.1, 0.1]\. LSTM is able to solve this problem, though by far not as fast as random weight guessing (see “Guessing” above). Clearly this trivial problem does not provide a very good testbed to compare performance of various nontrivial algorithms. Still, it demonstrates that LSTM does not encounter fundamental problems when faced with signal and noise on the same channel.

5.3.2 任务 3b。架构、参数及其他元素与任务 3a 相同，但现在将高斯噪声（均值为 0，方差为 0.2）添加到长短期记忆（LSTM）中。表 5：任务 3b：修改的两序列问题。编号 ST2：比例

5.3.2 Task 3b. The architecture, parameters, and other elements are as in task 3a, but now with Gaussian noise (mean 0 and variance 0.2) added to the Long Short-Term Memory 1759 Table 5: Task 3b: Modified Two-Sequence Problem. Number ST2: Fraction

信息传递元素（\t \le N\）。我们根据以下略微重新定义的标准停止训练（并判断任务是否解决）：ST1：来自随机选择测试集的 256 个序列中误分类的序列少于 6 个；ST2：满足 ST1，且测试集平均绝对误差低于 0.04。在 ST2 情况下，使用由 2560 个随机选择序列组成的额外测试集来确定误分类序列的比例。

Information-conveying elements (\t \le N\). We stop training (and judge the task as being solved) according to the following, slightly redefined criteria: ST1: fewer than 6 out of 256 sequences from a randomly chosen test set are misclassified; ST2: ST1 is satisfied, and mean absolute test set error is below 0.04. In case of ST2, an additional test set consisting of 2560 randomly chosen sequences is used to determine the fraction of misclassified sequences.

结果。见表 5。结果是 10 次不同权重初始化实验的均值。LSTM 轻松解决了该问题。

Results. See Table 5. The results represent means of 10 trials with different weight initializations. LSTM easily solves the problem.

5.3.3 任务 3c。架构、参数和其他元素与任务 3a 相同，但进行了一些关键改动使得任务不再简单：类别 1 和类别 2 的目标值分别为 0.2 和 0.8，并且目标值上添加了高斯噪声（均值 0，方差 0.1；标准差 0.32）。为了最小化均方误差，系统必须学习给定输入下目标值的条件期望。误分类定义为输出与无噪声目标（类别 1 为 0.2，类别 2 为 0.8）的绝对差大于 0.1。如果无噪声目标与输出的平均绝对差低于 0.015，则认为网络输出可接受。由于这需要高权重精度，任务 3c（不同于任务 3a 和 3b）无法通过随机猜测快速解决。

5.3.3 Task 3c. The architecture, parameters, and other elements are as in task 3a, but with a few essential changes that make the task nontrivial: the targets are 0.2 and 0.8 for class 1 and class 2, respectively, and there is Gaussian noise on the targets (mean 0 and variance 0.1; S.D. 0.32). To minimize mean squared error, the system has to learn the conditional expectations of the targets given the inputs. Misclassification is defined as absolute difference between output and noise-free target (0.2 for class 1 and 0.8 for class 2) > 0.1. The network output is considered acceptable if the mean absolute difference between noise-free target and output is below 0.015. Since this requires high weight precision, task 3c (unlike tasks 3a and 3b) cannot be solved quickly by random guessing.

训练/测试。学习率为 0.1。我们按以下标准停止训练：从随机选择的测试集中，256 个序列无一被误分类，且无噪声目标与输出的平均绝对差低于 0.015。另外使用由 2560 个随机选择序列组成的测试集来确定误分类序列的比例。

Training/Testing. The learning rate is 0.1. We stop training according to the following criterion: none of 256 sequences from a randomly chosen test set is misclassified, and mean absolute difference between the noise-free target and output is below 0.015. An additional test set consisting of 2560 randomly chosen sequences is used to determine the fraction of misclassified sequences.

结果。见表 6。结果是 10 次不同权重初始化实验的均值。尽管目标值带有噪声，LSTM 仍然可以通过学习期望目标值来解决问题。1760 Sepp Hochreiter 和 Jürgen Schmidhuber 表 6：任务 3c：修改后更具挑战性的双序列问题。数量 分数 平均差

Results. See Table 6. The results represent means of 10 trials with different weight initializations. Despite the noisy targets, LSTM still can solve the problem by learning the expected target values. 1760 Sepp Hochreiter and Jürgen Schmidhuber Table 6: Task 3c: Modified, More Challenging Two-Sequence Problem. Number Fraction Average Difference

5.4 实验 4：加法问题。本节中的困难任务属于其他循环网络算法从未解决过的类型。这表明 LSTM 能够解决涉及分布式连续值表示的长时间滞后问题。

5.4 Experiment 4: Adding Problem. The difficult task in this section is of a type that has never been solved by other recurrent net algorithms. It shows that LSTM can solve long-time-lag problems involving distributed, continuous-valued representations.

5.4.1 任务。每个输入序列的每个元素由两个分量组成。第一个分量是从区间 \([-1,1]\) 中随机选取的实数值；第二个分量为 1.0、0.0 或 -1.0，用作标记。在每个序列末尾，任务是对那些第二个分量等于 1.0 的配对的第一分量求和。序列长度在最小长度 T 和 \(T + T/10\) 之间随机。在给定序列中，恰好有两个配对被标记，具体如下：首先随机选择并标记前 10 个配对中的一个（其第一分量记为 \(X_1\)）。然后，从前 \(T/2 - 1\) 个尚未标记的配对中随机选择并标记一个（其第一分量记为 \(X_2\)）。除第一个和最后一个配对的第二分量为 -1 外，其余所有配对的第二分量均为零。（在极少数情况下，序列的第一个配对被标记，则将 \(X_1\) 设为零。）误差信号仅在序列末尾产生：目标为 \(0.5 + (X_1 + X_2)/4.0\)（即 \(X_1 + X_2\) 缩放到区间 \([0,1]\)）。当序列末尾的绝对误差低于 0.04 时，序列处理正确。

5.4.1 Task. Each element of each input sequence is a pair of components. The first component is a real value randomly chosen from the interval \([-1,1]\); the second is 1.0, 0.0, or -1.0 and is used as a marker. At the end of each sequence, the task is to output the sum of the first components of those pairs that are marked by second components equal to 1.0. Sequences have random lengths between the minimal sequence length T and \(T + T/10\). In a given sequence, exactly two pairs are marked, as follows: we first randomly select and mark one of the first 10 pairs (whose first component we call \(X_1\)). Then we randomly select and mark one of the first \(T/2 - 1\) still unmarked pairs (whose first component we call \(X_2\)). The second components of all remaining pairs are zero except for the first and final pair, whose second components are -1. (In the rare case where the first pair of the sequence gets marked, we set \(X_1\) to zero.) An error signal is generated only at the sequence end: the target is \(0.5 + (X_1 + X_2)/4.0\) (the sum \(X_1 + X_2\) scaled to the interval \([0,1]\)). A sequence is processed correctly if the absolute error at the sequence end is below 0.04.

5.4.2 架构。我们使用一个三层网络，有两个输入单元、一个输出单元和两个大小为 2 的细胞块。输出层仅从记忆细胞接收连接。记忆细胞和门单元从记忆细胞和门单元接收输入（隐藏层是全连接的；较少的连接也可能有效）。输入层前向连接到隐藏层中的所有单元。所有非输入单元都有偏置权重。这些架构参数使得存储至少两个输入信号变得容易（细胞块大小为 1 也同样有效）。所有激活函数均为逻辑函数，输出范围为 \([0,1]\)，但 \(h\) 的范围为 \([-1,1]\)，\(g\) 的范围为 \([-2,2]\)。

5.4.2 Architecture. We use a three-layer net with two input units, one output unit, and two cell blocks of size 2. The output layer receives connections only from memory cells. Memory cells and gate units receive inputs from memory cells and gate units (the hidden layer is fully connected; less connectivity may work as well). The input layer has forward connections to all units in the hidden layer. All noninput units have bias weights. These architecture parameters make it easy to store at least two input signals (a cell block size of 1 works well, too). All activation functions are logistic with output range \([0,1]\), except for \(h\), whose range is \([-1,1]\), and \(g\), whose range is \([-2,2]\).

5.4.3 状态漂移与初始偏置。注意，该任务需要长时间存储实数的精确值；系统必须学会保护记忆细胞内容免受即使微小的内部状态漂移（见第 4 节）。为了研究漂移问题的重要性，我们通过偏置所有非输入单元来使任务更加困难，从而人为地诱发内部状态漂移。所有权重（包括偏置权重）在区间 \([-0.1, 0.1]\) 内随机初始化。按照第 4 节中针对状态漂移的补救措施，第一个输入门偏置初始化为 -3.0，第二个输入门偏置初始化为

5.4.3 State Drift Versus Initial Bias. Note that the task requires storing the precise values of real numbers for long durations; the system must learn to protect memory cell contents against even minor internal state drift (see section 4). To study the significance of the drift problem, we make the task even more difficult by biasing all noninput units, thus artificially inducing internal state drift. All weights (including the bias weights) are randomly initialized in the range \([-0.1, 0.1]\). Following section 4's remedy for state drifts, the first input gate bias is initialized with -3.0 and the second with

-6.0（虽然精确值几乎无关紧要，附加实验也证实了这一点）。

-6.0 (though the precise values hardly matter, as confirmed by additional experiments).

5.4.4 训练/测试。学习率为 0.5。一旦平均训练误差低于 0.01 且最近 2000 个序列被正确处理后，训练停止。

5.4.4 Training/Testing. The learning rate is 0.5. Training is stopped once the average training error is below 0.01, and the 2000 most recent sequences were processed correctly.

5.4.5 实验结果。使用由 2560 个随机选择的序列组成的测试集，平均测试集误差始终低于 0.01，并且错误处理的序列从未超过三个。表 7 显示了细节。该实验表明，LSTM 能够很好地处理分布式表示，LSTM 能够学习执行涉及连续值的计算，并且由于系统能够在最小延迟 T/2 时间步内无退化地存储连续值，因此不存在显著的、有害的内部状态漂移。

5.4.5 Results. With a test set consisting of 2560 randomly chosen sequences, the average test set error was always below 0.01, and there were never more than three incorrectly processed sequences. Table 7 shows details. The experiment demonstrates that LSTM is able to work well with distributed representations, LSTM is able to learn to perform calculations involving continuous values, and since the system manages to store continuous values without deterioration for minimal delays of T/2 time steps, there is no significant, harmful internal state drift.

5.5 实验 5：乘法问题。有人可能会认为 LSTM 有点偏向于上一小节中的加法问题这样的任务。加法问题的解决方案可能利用了 CEC 内置的积分能力。虽然 CEC 的这一特性可以被视为一个特征而非缺点（积分似乎是现实世界中许多任务的自然子任务），但问题出现了：LSTM 是否也能解决本质上非积分性的任务？表 8：实验 5：乘法问题的结果。最小数量成功次数...

5.5 Experiment 5: Multiplication Problem. One may argue that LSTM is a bit biased toward tasks such as the adding problem from the previous subsection. Solutions to the adding problem may exploit the CEC's built-in integration capabilities. Although this CEC property may be viewed as a feature rather than a disadvantage (integration seems to be a natural subtask of many tasks occurring in the real world), the question arises whether LSTM can also solve tasks with inherently nonintegrative solutions. Table 8: Experiment 5: Results for the Multiplication Problem. Minimal Number of Number of Success

（积分似乎是现实世界中许多任务的自然子任务），那么问题出现了：LSTM 是否也能解决本质上非积分性的任务？为了测试这一点，我们修改了问题，要求最终目标等于先前标记输入的乘积（而不是和）。

feature rather than a disadvantage (integration seems to be a natural subtask of many tasks occurring in the real world), the question arises whether LSTM can also solve tasks with inherently nonintegrative solutions. To test this, we change the problem by requiring the final target to equal the product (instead of the sum) of earlier marked inputs.

5.5.1 任务。与 5.4 节中的任务类似，不同之处在于每对的第一个分量是从区间[0, 1]中随机选取的实数值。在极少见的情况下，输入序列的第一对被标记，我们将 X1 设为 1.0。序列末尾的目标是乘积 X1 × X2。

5.5.1 Task. This is like the task in section 5.4, except that the first component of each pair is a real value randomly chosen from the interval [0, 1]. In the rare case where the first pair of the input sequence gets marked, we set X1 to 1.0. The target at sequence end is the product X1 × X2.

5.5.2 架构。与 5.4 节相同。所有权重（包括偏置权重）在区间[−0.1, 0.1]内随机初始化。

5.5.2 Architecture. This is as in section 5.4. All weights (including the bias weights) are randomly initialized in the range [−0.1, 0.1].

5.5.3 训练/测试。学习率为 0.1。我们测试两次性能：当最近 2000 个训练序列中少于 nseq 个序列的绝对误差超过 0.04 时，即进行测试，其中 nseq = 140 和 nseq = 13。为什么选择这些值？nseq = 140 足以学习存储相关输入，但不足以微调精确的最终输出。而 nseq = 13 则产生相当令人满意的结果。

5.5.3 Training/Testing. The learning rate is 0.1. We test performance twice: as soon as less than nseq of the 2000 most recent training sequences lead to absolute errors exceeding 0.04, where nseq = 140 and nseq = 13. Why these values? nseq = 140 is sufficient to learn storage of the relevant inputs. It is not enough though to fine-tune the precise final outputs. nseq = 13, however, leads to quite satisfactory results.

5.5.4 结果。对于 nseq = 140（nseq = 13），使用由 2560 个随机选择序列组成的测试集，平均测试集误差始终低于 0.026（0.013），错误处理的序列从未超过 170（15）个。表 8 显示了详细信息。（带有额外标准隐藏单元或记忆单元上方隐藏层的网络可能更快地学习微调部分。）该实验表明，LSTM 可以解决同时涉及连续值表示和非整合信息处理的任务。Long Short-Term Memory 1763

5.5.4 Results. For nseq = 140 (nseq = 13) with a test set consisting of 2560 randomly chosen sequences, the average test set error was always below 0.026 (0.013), and there were never more than 170 (15) incorrectly processed sequences. Table 8 shows details. (A net with additional standard hidden units or with a hidden layer above the memory cells may learn the fine-tuning part more quickly.) The experiment demonstrates that LSTM can solve tasks involving both continuous-valued representations and nonintegrative information processing. Long Short-Term Memory 1763

5.6 实验 6：时序顺序。在本小节中，LSTM 解决了以前循环网络算法从未解决过的其他困难（但人为的）任务。实验表明，LSTM 能够提取由广泛分离输入的时序顺序所传达的信息。

5.6 Experiment 6: Temporal Order. In this subsection, LSTM solves other difficult (but artificial) tasks that have never been solved by previous recurrent net algorithms. The experiment shows that LSTM is able to extract information conveyed by the temporal order of widely separated inputs.

5.6.1 任务 6a：两个相关且广泛分离的符号。目标是对序列进行分类。元素和目标采用局部表示（只有一个非零位的输入向量）。序列以 E 开始，以 B（“触发符号”）结束，其余部分由从集合{a, b, c, d}中随机选择的符号组成，除了在位置 t1 和 t2 的两个元素是 X 或 Y。序列长度在 100 到 110 之间随机选择，t1 在 10 到 20 之间随机选择，t2 在 50 到 60 之间随机选择。共有四类序列：

5.6.1 Task 6a: Two Relevant, Widely Separated Symbols. The goal is to classify sequences. Elements and targets are represented locally (input vectors with only one nonzero bit). The sequence starts with an E, ends with a B (the “trigger symbol”), and otherwise consists of randomly chosen symbols from the set {a, b, c, d} except for two elements at positions t1 and t2 that are either X or Y. The sequence length is randomly chosen between 100 and 110, t1 is randomly chosen between 10 and 20, and t2 is randomly chosen between 50 and 60. There are four sequence classes

Q, R, S, U，它们取决于 X 和 Y 的时序顺序。规则如下：

Q, R, S, U, which depend on the temporal order of X and Y. The rules are:

5.6.2 任务 6b：三个相关且广泛分离的符号。同样，目标是分类序列。元素和目标采用局部表示。序列以 E 开始，以 B（触发符号）结束，其余部分由从集合{a, b, c, d}中随机选择的符号组成，除了在位置 t1、t2 和 t3 处的三个元素，它们要么是 X 要么是 Y。序列长度在 100 到 110 之间随机选择，t1 在 10 到 20 之间随机选择，t2 在 33 到 43 之间随机选择，t3 在 66 到 76 之间随机选择。有八个序列类别——

5.6.2 Task 6b: Three Relevant, Widely Separated Symbols. Again, the goal is to classify sequences. Elements and targets are represented locally. The sequence starts with an E, ends with a B (the trigger symbol), and otherwise consists of randomly chosen symbols from the set {a, b, c, d} except for three elements at positions t1, t2, and t3 that are either X or Y. The sequence length is randomly chosen between 100 and 110, t1 is randomly chosen between 10 and 20, t2 is randomly chosen between 33 and 43, and t3 is randomly chosen between 66 and 76. There are eight sequence classes—

Q、R、S、U、V、A、B、C——取决于 X 和

Q, R, S, U, V, A, B, C—which depend on the temporal order of the Xs and

Ys。规则如下：X, X, X → Q; X, X, Y → R; X, Y, X → S; X, Y, Y →

Ys. The rules are: X, X, X → Q; X, X, Y → R; X, Y, X → S; X, Y, Y →

U; Y, X, X → V; Y, X, Y → A; Y, Y, X → B; Y, Y, Y → C。输出单元的数量与类别数相同。每个类别由一个二进制目标向量局部表示，该向量有一个非零分量。对于两个任务，误差信号仅在序列结束时出现。如果所有输出单元最终绝对误差低于 0.3，则序列被正确分类。

U; Y, X, X → V; Y, X, Y → A; Y, Y, X → B; Y, Y, Y → C. There are as many output units as there are classes. Each class is locally represented by a binary target vector with one nonzero component. With both tasks, error signals occur only at the end of a sequence. The sequence is classified correctly if the final absolute error of all output units is below 0.3.

架构。我们使用一个三层网络，有八个输入单元，两个（三个）大小为 2 的细胞块，以及任务 6a（6b）的四个（八个）输出单元。同样，所有非输入单元都有偏置权重，输出层仅从记忆细胞接收连接。记忆细胞和门控单元接收来自输入单元、记忆细胞和门控单元的输入（隐藏层是全连接的；较少连接也可能有效）。任务 6a（6b）的架构参数使得至少存储两个（三个）输入信号变得容易。所有激活函数都是逻辑函数，输出范围为[0,1]，但 h 的输出范围为[-1,1]，g 的输出范围为[-2,2]。1764 Sepp Hochreiter 和 Jürgen Schmidhuber 表 9：实验 6：时序问题的结果。任务 权重数量 错误预测数 成功后的训练序列数 任务 6a 156 1/2560 31,390 任务 6b 308 2/2560 571,100 注：“错误预测数”是测试集（含 2560 个序列）中错误分类的序列数（至少一个输出单元误差>0.3）。最右列给出了达到停止标准所需的训练序列数。任务 6a 的结果是 20 次试验的平均值；任务 6b 是 10 次试验的平均值。

Architecture. We use a three-layer net with eight input units, two (three) cell blocks of size 2, and four (eight) output units for task 6a (6b). Again all noninput units have bias weights, and the output layer receives connections from memory cells only. Memory cells and gate units receive inputs from input units, memory cells, and gate units (the hidden layer is fully connected; less connectivity may work as well). The architecture parameters for task 6a (6b) make it easy to store at least two (three) input signals. All activation functions are logistic with output range [0, 1], except for h, whose range is [-1, 1], and g, whose range is [-2, 2]. 1764 Sepp Hochreiter and Jürgen Schmidhuber Table 9: Experiment 6: Results for the Temporal Order Problem. Number of Number of Task Weights Wrong Predictions Success After Task 6a 156 1 out of 2560 31,390 Task 6b 308 2 out of 2560 571,100 Notes: “Number of Wrong Predictions” is the number of incorrectly classified sequences (error > 0.3 for at least one output unit) from a test set containing 2560 sequences. The right-most column gives the number of training sequences required to achieve the stopping criterion. The results for task 6a are means of 20 trials; those for task 6b of 10 trials.

训练/测试。实验 6a（6b）的学习率为 0.5（0.1）。当平均训练误差降至 0.1 以下且最近 2000 个序列均被正确分类时，停止训练。所有权重在[-0.1, 0.1]范围内初始化。第一个输入门偏置初始化为-2.0，第二个为-4.0，实验 6b 中第三个为-6.0（附加实验证实精确值几乎无关紧要）。

Training/Testing. The learning rate is 0.5 (0.1) for experiment 6a (6b). Training is stopped once the average training error falls below 0.1 and the 2000 most recent sequences were classified correctly. All weights are initialized in the range [-0.1, 0.1]. The first input gate bias is initialized with -2.0, the second with -4.0, and (for experiment 6b) the third with -6.0 (again, we confirmed by additional experiments that the precise values hardly matter).

结果。使用由 2560 个随机选择序列组成的测试集，平均测试集误差始终低于 0.1，且错误分类的序列从未超过三个。表 9 显示了详细信息。实验表明，LSTM 能够从广泛分离输入的时间顺序中提取信息。例如，在任务 6a 中，第一个和第二个相关输入之间以及第二个相关输入与序列结束之间的延迟至少为 30 个时间步。

Results. With a test set consisting of 2560 randomly chosen sequences, the average test set error was always below 0.1, and there were never more than three incorrectly classified sequences. Table 9 shows details. The experiment shows that LSTM is able to extract information conveyed by the temporal order of widely separated inputs. In task 6a, for instance, the delays between the first and second relevant input and between the second relevant input and sequence end are at least 30 time steps.

典型解决方案。在实验 6a 中，LSTM 如何区分时间顺序（X, Y）和（Y, X）？众多可能解之一是将第一个 X 或 Y 存储在细胞块 1 中，将第二个 X/Y 存储在细胞块 2 中。在第一个 X/Y 出现之前，块 1 通过其循环连接感知到自身为空。第一个 X/Y 之后，块 1 关闭其输入门。一旦块 1 被填满并关闭，这一事实将对块 2 可见（回忆所有门单元和所有记忆细胞接收来自所有非输出单元的连接）。然而，典型解仅需一个记忆细胞块。该块存储第一个 X 或 Y；当第二个 X/Y 出现时，其状态根据第一个存储符号改变。解类型 1 利用记忆细胞输出与输入门单元之间的连接。以下事件导致不同的输入门激活：X 与已填充块同时出现；X 与空块同时出现。解类型 2 基于记忆细胞输出与记忆细胞输入之间的强正连接。X（Y）的先前出现由正（负）内部状态表示。一旦输入门第二次打开，输出门也随之打开，记忆细胞输出反馈到自身输入。这导致（X, Y）由正内部状态表示，因为 X（通过当前内部状态和细胞输出反馈）两次贡献于新内部状态。类似地，（Y, X）由负内部状态表示。

Typical Solutions. In experiment 6a, how does LSTM distinguish between temporal orders (X, Y) and (Y, X)? One of many possible solutions is to store the first X or Y in cell block 1 and the second X/Y in cell block 2. Before the first X/Y occurs, block 1 can see that it is still empty by means of its recurrent connections. After the first X/Y, block 1 can close its input gate. Once block 1 is filled and closed, this fact will become visible to block 2 (recall that all gate units and all memory cells receive connections from all nonoutput units). Typical solutions, however, require only one memory cell block. The block stores the first X or Y; once the second X/Y occurs, it changes its state depending on the first stored symbol. Solution type 1 exploits the connection between memory cell output and input gate unit. The following events cause different input gate activations: X occurs in conjunction with a filled block; X occurs in conjunction with an empty block. Solution type 2 is based on a strong, positive connection between memory cell output and memory cell input. The previous occurrence of X (Y) is represented by a positive (negative) internal state. Once the input gate opens for the second time, so does the output gate, and the memory cell output is fed back to its own input. This causes (X, Y) to be represented by a positive internal state, because X contributes to the new internal state twice (via current internal state and cell output feedback). Similarly, (Y, X) gets represented by a negative internal state.

注释：第 1 列：任务编号。第 2 列：最小序列长度 p。第 3 列：最近相关输入信息与教师信号之间的最小步数。第 4 列：细胞块数 b。第 5 列：块大小 s。第 6 列：输入单元数 in。第 7 列：输出单元数 out。第 8 列：权重数 w。第 9 列：c 描述连接性：F 表示“输出层接收来自记忆细胞的连接；记忆细胞和门单元接收来自输入单元、记忆细胞和门单元的连接”；B 表示“每一层接收来自其下所有层的连接”。第 10 列：初始输出门偏置 ogb，其中 r 表示“从区间[-0.1, 0.1]随机选择”，no og 表示“未使用输出门”。第 11 列：初始输入门偏置 igb（参见第 10 列）。第 12 列：哪些单元具有偏置权重？

Notes: Col. 1: task number. Col. 2: minimal sequence length p. Col. 3: minimal number of steps between most recent relevant input information and teacher signal. Col. 4: number of cell blocks b. Col. 5: block size s. Col. 6: Number of input units in. Col. 7: Number of output units out. Col. 8: number of weights w. Col. 9: c describes connectivity: F means “output layer receives connections from memory cells; memory cells and gate units receive connections from input units, memory cells and gate units”; B means “each layer receives connections from all layers below.” Col. 10: Initial output gate bias ogb, where r stands for “randomly chosen from the interval [-0.1, 0.1]” and no og means “no output gate used.” Col. 11: initial input gate bias igb (see Col. 10). Col. 12: which units have bias weights?

b1 表示“所有隐藏单元”，ga 表示“仅门单元”，all 表示“所有非输入单元”。第 13 列：函数 h，其中 id 为恒等函数，h1 为[-2, 2]范围内的逻辑 S 型函数。第 14 列：逻辑函数 g，其中 g1 为[0, 1]范围内的 S 型函数，g2 为[-1, 1]范围内的 S 型函数。第 15 列：学习率\(\alpha\)。正（负）内部状态。一旦输入门第二次打开，输出门也随之打开，记忆细胞输出反馈到自身输入。这导致（X, Y）由正内部状态表示，因为 X（通过当前内部状态和细胞输出反馈）两次贡献于新内部状态。类似地，（Y, X）由负内部状态表示。

b1 stands for “all hidden units”, ga for “only gate units,” and all for “all noninput units.” Col. 13: the function h, where id is identity function, h1 is logistic sigmoid in [-2, 2]. Col. 14: the logistic function g, where g1 is sigmoid in [0, 1], g2 in [-1, 1]. Col. 15: learning rate \(\alpha\). Positive (negative) internal state. Once the input gate opens for the second time, so does the output gate, and the memory cell output is fed back to its own input. This causes (X, Y) to be represented by a positive internal state, because X contributes to the new internal state twice (via current internal state and cell output feedback). Similarly, (Y, X) gets represented by a negative internal state.

5.7 实验条件总结。表 10 和表 11 概述了实验 1 至 6 最重要的 LSTM 参数和架构细节。由于历史原因，简单实验 2a 和 2b 的条件与其他更系统的实验略有不同。表 11：LSTM 实验条件总结，第二部分。(1) (2) (3) (4) (5) (6) 任务 选择 区间 测试集大小 停止准则 成功 1 t1 [−0.2, 0.2] 256 训练和测试正确预测 见正文 2a t1 [−0.2, 0.2] 无测试集 经过 500 万样本 ABS(0.25) 2b t2 [−0.2, 0.2] 10,000 经过 500 万样本 ABS(0.25) 2c t2 [−0.2, 0.2] 10,000 经过 500 万样本 ABS(0.2) 3a t3 [−0.1, 0.1] 2560 ST1 和 ST2（见正文） ABS(0.2) 3b t3 [−0.1, 0.1] 2560 ST1 和 ST2（见正文） ABS(0.2) 3c t3 [−0.1, 0.1] 2560 ST1 和 ST2（见正文） 见正文 4 t3 [−0.1, 0.1] 2560 ST3(0.01) ABS(0.04) 5 t3 [−0.1, 0.1] 2560 见正文 ABS(0.04) 6a t3 [−0.1, 0.1] 2560 ST3(0.1) ABS(0.3) 6b t3 [−0.1, 0.1] 2560 ST3(0.1) ABS(0.3) 注释：第 1 列：任务编号。第 2 列：训练样本选择，其中 t1 表示“从训练集中随机选择”，t2 表示“从两个类别中随机选择”，t3 表示“在线随机生成”。第 3 列：权重初始化区间。第 4 列：测试集大小。第 5 列：训练停止准则，其中 ST3(\beta\)表示“平均训练误差低于\beta\且最近 2000 个序列被正确处理”。第 6 列：成功（正确分类）准则，其中 ABS(\beta\)表示“序列结束时所有输出单元的绝对误差低于\beta\”。

5.7 Summary of Experimental Conditions. Tables 10 and 11 provide an overview of the most important LSTM parameters and architectural details for experiments 1 through 6. The conditions of the simple experiments 2a and 2b differ slightly from those of the other, more systematic experiments, due to historical reasons. Table 11: Summary of Experimental Conditions for LSTM, Part II. (1) (2) (3) (4) (5) (6) Task Select Interval Test Set Size Stopping Criterion Success 1 t1 [−0.2, 0.2] 256 Training and test correctly pred. See text 2a t1 [−0.2, 0.2] no test set After 5 million exemplars ABS(0.25) 2b t2 [−0.2, 0.2] 10,000 After 5 million exemplars ABS(0.25) 2c t2 [−0.2, 0.2] 10,000 After 5 million exemplars ABS(0.2) 3a t3 [−0.1, 0.1] 2560 ST1 and ST2 (see text) ABS(0.2) 3b t3 [−0.1, 0.1] 2560 ST1 and ST2 (see text) ABS(0.2) 3c t3 [−0.1, 0.1] 2560 ST1 and ST2 (see text) See text 4 t3 [−0.1, 0.1] 2560 ST3(0.01) ABS(0.04) 5 t3 [−0.1, 0.1] 2560 see text ABS(0.04) 6a t3 [−0.1, 0.1] 2560 ST3(0.1) ABS(0.3) 6b t3 [−0.1, 0.1] 2560 ST3(0.1) ABS(0.3) Notes: Col. 1: task number. Col. 2: training exemplar selection, where t1 stands for "randomly chosen from training set," t2 for "randomly chosen from two classes," and t3 for "randomly generated on line." Col. 3: weight initialization interval. Col. 4: test set size. Col. 5: Stopping criterion for training, where ST3(\beta\) stands for "average training error below \beta\ and the 2000 most recent sequences were processed correctly." Col. 6: success (correct classification) criterion, where ABS(\beta\) stands for "absolute error of all output units at sequence end is below \beta\."

• 特别高效的截断反向传播版本的 LSTM 算法不容易解决类似于强延迟 XOR 问题的问题，其中目标是计算两个在噪声序列中先前出现的相距很远的输入的 XOR。原因在于仅存储其中一个输入无助于减少期望误差；该任务是不可分解的，即无法通过先解决一个较简单的子目标来逐步减少误差。理论上，可以通过使用完整梯度（也许加上从记忆细胞接收输入的额外常规隐藏单元）来规避这一限制。但我们不推荐计算完整梯度，原因如下：(1) 它增加了计算复杂度；(2) 通过 CEC 的恒定误差流仅对截断 LSTM 成立；(3) 我们实际上确实进行了一些非截断 LSTM 的实验。与截断 LSTM 相比没有显著差异，正是因为 CEC 之外，误差流往往会迅速消失。出于同样的原因，完整 BPTT 并不优于截断 BPTT。

• The particularly efficient truncated backpropagation version of the LSTM algorithm will not easily solve problems similar to strongly delayed XOR problems, where the goal is to compute the XOR of two widely separated inputs that previously occurred somewhere in a noisy sequence. The reason is that storing only one of the inputs will not help to reduce the expected error; the task is nondecomposable in the sense that it is impossible to reduce the error incrementally by first solving an easier subgoal. In theory, this limitation can be circumvented by using the full gradient (perhaps with additional conventional hidden units receiving input from the memory cells). But we do not recommend computing the full gradient for the following reasons: (1) It increases computational complexity, (2) constant error flow through CECs can be shown only for truncated LSTM, and (3) we actually did conduct a few experiments with non-truncated LSTM. There was no significant difference to truncated LSTM, exactly because outside the CECs, error flow tends to vanish quickly. For the same reason, full BPTT does not outperform truncated BPTT.

• 每个记忆细胞块需要两个额外的单元（输入门和输出门）。然而，与标准循环网络相比，这并不会将权重数量增加超过 9 倍：每个传统隐藏单元在 LSTM 架构中最多被三个单元取代，在全连接情况下权重数量增加一个因子\(3^2\)（即 9 倍）。但请注意，我们的实验中 LSTM 和竞争方法的架构使用了相当可比的权重数量。

• Each memory cell block needs two additional units (input and output gate). In comparison to standard recurrent nets, however, this does not increase the number of weights by more than a factor of 9: each conventional hidden unit is replaced by at most three units in the LSTM architecture, increasing the number of weights by a factor of \(3^2\) (i.e., 9) in the fully connected case. Note, however, that our experiments use quite comparable weight numbers for the architectures of LSTM and competing approaches.

• 由于通过记忆细胞内的 CEC 保持恒定误差流，LSTM 通常会遇到与前馈网络一次性看到整个输入字符串类似的问题。例如，有些任务可以通过随机权重猜测快速解决，但截断 LSTM 算法在小的权重初始化下却无法解决，比如 500 步奇偶校验问题（见第 5 节引言）。这里，LSTM 的问题类似于一个有 500 个输入、试图解决 500 位奇偶校验的前馈网络。事实上，LSTM 的表现很像一个由反向传播训练、看到整个输入的前馈网络。但这正是它为何在具有显著搜索空间的许多非平凡任务上明显优于先前方法的原因。

• Due to its constant error flow through CECs within memory cells, LSTM generally runs into problems similar to those of feedforward nets' seeing the entire input string at once. For instance, there are tasks that can be quickly solved by random weight guessing but not by the truncated LSTM algorithm with small weight initializations, such as the 500-step parity problem (see the introduction to section 5). Here, LSTM's problems are similar to the ones of a feedforward net with 500 inputs, trying to solve 500-bit parity. Indeed LSTM typically behaves much like a feedforward net trained by backpropagation that sees the entire input. But that is also precisely why it so clearly outperforms previous approaches on many nontrivial tasks with significant search spaces.

• LSTM 在近因性概念上并不存在超出其他方法的问题。然而，所有基于梯度的方法都实际无法精确计数离散时间步长。如果某个信号是发生在 99 步前还是 100 步前有所区别，那么似乎需要一个额外的计数机制。但较简单的任务，例如仅需区分 3 步和 11 步的情况，对 LSTM 不构成任何问题。例如，通过在记忆细胞输出和输入之间产生适当的负连接，LSTM 可以给近期输入更大权重，并在必要时学习衰减。

• LSTM does not have any problems with the notion of recency that go beyond those of other approaches. All gradient-based approaches, however, suffer from a practical inability to count discrete time steps precisely. If it makes a difference whether a certain signal occurred 99 or 100 steps ago, then an additional counting mechanism seems necessary. Easier tasks, however, such as one that requires making a difference only between, say, 3 and 11 steps, do not pose any problems to LSTM. For instance, by generating an appropriate negative connection between memory cell output and input, LSTM can give more weight to recent inputs and learn decays where necessary.

• 记忆单元内的恒定误差反向传播使得 LSTM 能够在处理与上述问题类似的问题时桥接非常长的时间滞后。

• The constant error backpropagation within memory cells results in LSTM’s ability to bridge very long time lags in case of problems similar to those discussed above.

• 对于诸如本文讨论的长时滞问题，LSTM 能够处理噪声、分布式表示和连续值。与有限状态自动机或隐马尔可夫模型相比，LSTM 不需要先验选择有限数量的状态。原则上，它可以处理无限的状态数。

• For long-time-lag problems such as those discussed in this article, LSTM can handle noise, distributed representations, and continuous values. In contrast to finite state automata or hidden Markov models, LSTM does not require an a priori choice of a finite number of states. In principle, it can deal with unlimited state numbers.

• 对于本文讨论的问题，LSTM 具有良好的泛化能力，即使输入序列中相隔较远的相关输入的位置无关紧要。与先前的方法不同，我们的方法能够快速学会区分输入序列中同一元素的两个或多个相隔较远的出现，而不依赖于适当的短时滞训练样本。

• For problems discussed in this article, LSTM generalizes well, even if the positions of widely separated, relevant inputs in the input sequence do not matter. Unlike previous approaches, ours quickly learns to distinguish between two or more widely separated occurrences of a particular element in an input sequence, without depending on appropriate short-time-lag training exemplars.

• 似乎不需要进行参数微调。LSTM 在广泛参数范围内（如学习率、输入门偏置和输出门偏置）都能良好工作。例如，对于一些读者来说，我们实验中使用的学习率可能看起来很大。然而，大的学习率会将输出门推向零，从而自动抵消其自身的负面影响。

• There appears to be no need for parameter fine-tuning. LSTM works well over a broad range of parameters such as learning rate, input gate bias, and output gate bias. For instance, to some readers the learning rates used in our experiments may seem large. However, a large learning rate pushes the output gates toward zero, thus automatically countermanding its own negative effects.

• LSTM 算法每个权重和时间步的更新复杂度本质上与 BPTT 相同，即 O(1)。与 RTRL 等其他方法相比，这非常优秀。然而，与完全 BPTT 不同，LSTM 在空间和时间上都是局部的。

• The LSTM algorithm’s update complexity per weight and time step is essentially that of BPTT, namely, O(1). This is excellent in comparison to other approaches such as RTRL. Unlike full BPTT, however, LSTM is local in both space and time.

每个记忆单元的内部架构保证其 CEC 内误差流恒定，前提是截断反向传播切断试图泄漏出记忆单元的误差流。这代表了连接极长时滞的基础。两个门单元学会打开和关闭每个记忆单元 CEC 内误差流的访问。乘法输入门保护 CEC 免受无关输入的扰动。类似地，乘法输出门保护其他单元免受当前无关记忆内容的扰动。为了了解 LSTM 的实际局限性，我们打算将其应用于真实世界数据。应用领域包括时间序列预测、音乐作曲和语音处理。使用 LSTM 增强序列分块器（Schmidhuber, 1992b, 1993）以结合两者的优势也将是有趣的。

Each memory cell’s internal architecture guarantees constant error flow within its CEC, provided that truncated backpropagation cuts off error flow trying to leak out of memory cells. This represents the basis for bridging very long time lags. Two gate units learn to open and close access to error flow within each memory cell’s CEC. The multiplicative input gate affords protection of the CEC from perturbation by irrelevant inputs. Similarly, the multiplicative output gate protects other units from perturbation by currently irrelevant memory contents. To find out about LSTM’s practical limitations we intend to apply it to real-world data. Application areas will include time-series prediction, music composition, and speech processing. It will also be interesting to augment sequence chunkers (Schmidhuber, 1992b, 1993) by LSTM to combine the advantages of both.

附录 A.1 算法细节。在下文中，下标\(k\)遍历输出单元，\(i\)遍历隐藏单元，\(c_j\)表示第\(j\)个记忆单元块，\(c_{vj}\)

Appendix A.1 Algorithm Details. In what follows, the index \(k\) ranges over output units, \(i\) ranges over hidden units, \(c_j\) stands for the \(j\)th memory cell block, \(c_{vj}\)

表示记忆单元块\(c_j\)的第\(v\)个单元，\(u, l, m\)表示任意单元，\(t\)遍历给定输入序列的所有时间步。实验中使用的门单元逻辑 Sigmoid 函数（范围\([0, 1]\)）为

denotes the \(v\)th unit of memory cell block \(c_j\), \(u, l, m\) stand for arbitrary units, and \(t\) ranges over all time steps of a given input sequence. The gate unit logistic sigmoid (with range \([0 , 1]\)) used in the experiments is

\(f(x) = \frac{1}{1 + \exp(-x)}\)。（A.1）Long Short-Term Memory 1769 实验中使用的函数\(h\)（范围\([-1, 1]\)）为

\(f(x) = \frac{1}{1 + \exp(-x)}\). (A.1) Long Short-Term Memory 1769 The function \(h\) (with range \([-1, 1]\)) used in the experiments is

\(h(x) = \frac{2}{1 + \exp(-x)} - 1\)。（A.2）实验中使用的函数\(g\)（范围\([-2, 2]\)）为

\(h(x) = \frac{2}{1 + \exp(-x)} - 1\). (A.2) The function \(g\) (with range \([-2, 2]\)) used in the experiments is

A.1.1 前向传播。隐藏单元 i 的净输入和激活

A.1.1 Forward Pass. The net input and the activation of hidden unit i

\(y_i(t) = f_i(\text{net}_i(t))\)。输入单元 j 的净输入和激活为：

\(y_i(t) = f_i(\text{net}_i(t))\). The net input and the activation of input unit j are:

\(y_{\text{in}_j}(t) = f_{\text{in}_j}(\text{net}_{\text{in}_j}(t))\)。输出单元 j 的净输入和激活为：

\(y_{\text{in}_j}(t) = f_{\text{in}_j}(\text{net}_{\text{in}_j}(t))\). The net input and the activation of output unit j are:

\(y_{\text{out}_j}(t) = f_{\text{out}_j}(\text{net}_{\text{out}_j}(t))\)。记忆单元块 cj 中第 v 个记忆单元的净输入 \(\text{net}_{c,vj}\)、内部状态 \(s_{c,vj}\) 和输出激活 \(y_{c,vj}\) 为：

\(y_{\text{out}_j}(t) = f_{\text{out}_j}(\text{net}_{\text{out}_j}(t))\). The net input \(\text{net}_{c,vj}\), the internal state \(s_{c,vj}\), and the output activation \(y_{c,vj}\) of the v-th memory cell of memory cell block cj are:

\(y_{c,vj}(t) = y_{\text{out}_j}(t) h(s_{c,vj}(t))\)。输出单元 k 的净输入和激活为：

\(y_{c,vj}(t) = y_{\text{out}_j}(t) h(s_{c,vj}(t))\). The net input and the activation of output unit k are:

y_k(t) = f_k(\text{net}_k(t))。后面要描述的反向传播基于以下截断反向传播公式。[1770] Sepp Hochreiter 和 Jürgen Schmidhuber

y_k(t) = f_k(\text{net}_k(t)). The backward pass to be described later is based on the following truncated backpropagation formulas. [1770] Sepp Hochreiter and Jürgen Schmidhuber

A.1.2 截断反向传播的近似导数。截断版本（见第 4 节）仅近似偏导数，这在下文中用≈tr 符号表示。它一旦误差离开记忆单元或门单元，就截断误差流。截断确保没有环路使得通过输入或输入门离开某个记忆单元的误差能通过其输出或输出门重新进入该单元。这反过来确保了通过记忆单元 CEC 的恒定误差流。在截断反向传播版本中，以下导数被替换为零：

A.1.2 Approximate Derivatives for Truncated Backpropagation. The truncated version (see section 4) only approximates the partial derivatives, which is reflected by the ≈tr signs in the notation below. It truncates error flow once it leaves memory cells or gate units. Truncation ensures that there are no loops across which an error that left some memory cell through its input or input gate can reenter the cell through its output or output gate. This in turn ensures constant error flow through the memory cell’s CEC. In the truncated backpropagation version, the following derivatives are replaced by zero:

\[ \frac{\partial \text{net}_{in_j}(t)}{\partial y_u(t-1)} \approx_{tr} 0 \quad \forall u, \qquad \frac{\partial \text{net}_{out_j}(t)}{\partial y_u(t-1)} \approx_{tr} 0 \quad \forall u, \]

\[ \frac{\partial \text{net}_{in_j}(t)}{\partial y_u(t-1)} \approx_{tr} 0 \quad \forall u, \qquad \frac{\partial \text{net}_{out_j}(t)}{\partial y_u(t-1)} \approx_{tr} 0 \quad \forall u, \]

\[ f'(\text{net}_{in_j}(t)) \frac{\partial \text{net}_{in_j}(t)}{\partial y_u(t-1)} \approx_{tr} 0 \quad \forall u, \qquad \frac{\partial y_{out_j}(t)}{\partial y_u(t-1)} = f'(\text{net}_{out_j}(t)) \frac{\partial \text{net}_{out_j}(t)}{\partial y_u(t-1)} \approx_{tr} 0 \quad \forall u, \]

\[ f'(\text{net}_{in_j}(t)) \frac{\partial \text{net}_{in_j}(t)}{\partial y_u(t-1)} \approx_{tr} 0 \quad \forall u, \qquad \frac{\partial y_{out_j}(t)}{\partial y_u(t-1)} = f'(\text{net}_{out_j}(t)) \frac{\partial \text{net}_{out_j}(t)}{\partial y_u(t-1)} \approx_{tr} 0 \quad \forall u, \]

\[ f'(\text{net}_{out_j}(t)) \frac{\partial \text{net}_{out_j}(t)}{\partial y_u(t-1)} \approx_{tr} 0 \quad \forall u, \]

\[ f'(\text{net}_{out_j}(t)) \frac{\partial \text{net}_{out_j}(t)}{\partial y_u(t-1)} \approx_{tr} 0 \quad \forall u, \]

\frac{\partial y_{c_j}(t)}{\partial y_u(t-1)} = \frac{\partial y_{c_j}(t)}{\partial \mathrm{net}_{\mathrm{out}_j}(t)} \frac{\partial \mathrm{net}_{\mathrm{out}_j}(t)}{\partial y_u(t-1)} + \frac{\partial y_{c_j}(t)}{\partial \mathrm{net}_{\mathrm{in}_j}(t)} \frac{\partial \mathrm{net}_{\mathrm{in}_j}(t)}{\partial y_u(t-1)}

\frac{\partial y_{c_j}(t)}{\partial y_u(t-1)} = \frac{\partial y_{c_j}(t)}{\partial \mathrm{net}_{\mathrm{out}_j}(t)} \frac{\partial \mathrm{net}_{\mathrm{out}_j}(t)}{\partial y_u(t-1)} + \frac{\partial y_{c_j}(t)}{\partial \mathrm{net}_{\mathrm{in}_j}(t)} \frac{\partial \mathrm{net}_{\mathrm{in}_j}(t)}{\partial y_u(t-1)}

+ \frac{\partial y_{c_j}(t)}{\partial \mathrm{net}_{c_j}(t)} \frac{\partial \mathrm{net}_{c_j}(t)}{\partial y_u(t-1)} \approx 0, \forall u.

+ \frac{\partial y_{c_j}(t)}{\partial \mathrm{net}_{c_j}(t)} \frac{\partial \mathrm{net}_{c_j}(t)}{\partial y_u(t-1)} \approx 0, \forall u.

这意味着对所有不在连接到 c_j、in_j、out_j 上的权重 w_{lm}（即 l \notin {c_j, in_j, out_j}）：

This implies for all w_{lm} not on connections to c_j, in_j, out_j (that is, l \notin {c_j, in_j, out_j}):

输出单元 k 的截断导数为：\frac{\partial y_i(t-1)}{\partial w_{lm}} otherwise (A.8)，其中 δ 是克罗内克δ函数（δ_{ab}=1 若 a=b，否则为 0），S_j 是记忆单元块 c_j 的大小。隐藏单元的截断导数

The truncated derivatives of output unit k are: \frac{\partial y_i(t-1)}{\partial w_{lm}} otherwise (A.8) where δ is the Kronecker delta (δ_{ab}=1 if a=b and 0 otherwise), and S_j is the size of memory cell block c_j. The truncated derivatives of a hidden unit

∂y cj (t)∂yu(t − 1) = ∂y cj (t)∂net out j (t)∂net out j (t)∂y u(t − 1) + ∂y cj (t)∂net in j (t)∂net in j (t)∂yu(t − 1)

\(net_i(t)) y^m(t-1)\。(A.9)（此处虽可使用完整梯度，但并不会影响通过记忆单元内部状态的恒定误差流。）细胞块 cj 的截断导数为：

\(net_i(t)) y^m(t-1)\. (A.9) (Here it would be possible to use the full gradient without affecting constant error flow through internal states of memory cells.) Cell block cj's truncated derivatives are:

)\\frac{\partial net_{cvj}(t)}{\partial w_{lm}}\ 1772 Sepp Hochreiter 和 J. Schmidhuber

)\\frac{\partial net_{cvj}(t)}{\partial w_{lm}}\ 1772 Sepp Hochreiter and J. Schmidhuber

\= \frac{\partial y_{out_j}(t)}{\partial w_{lm}} h(s_{cvj}(t)) + h'(s_{cvj}(t)) \frac{\partial s_{cvj}(t)}{\partial w_{lm}} y_{out_j}(t)\

\= \frac{\partial y_{out_j}(t)}{\partial w_{lm}} h(s_{cvj}(t)) + h'(s_{cvj}(t)) \frac{\partial s_{cvj}(t)}{\partial w_{lm}} y_{out_j}(t)\

\h'(s_{cvj}(t)) \frac{\partial s_{cvj}(t)}{\partial w_{lm}} y_{out_j}(t)\。(A.13) 为了在时刻 t 高效更新系统，需要在时刻 t−1 存储的唯一（截断）导数为

\h'(s_{cvj}(t)) \frac{\partial s_{cvj}(t)}{\partial w_{lm}} y_{out_j}(t)\. (A.13) To update the system efficiently at time t, the only (truncated) derivatives that need to be stored at time t − 1 are

A.1.3 反向传播。我们将仅针对 LSTM 算法中特别高效的截断梯度版本描述反向传播。为简便起见，即便根据上述截断反向传播方程进行了近似，我们仍使用等号。时刻 t 的平方误差由下式给出：

A.1.3 Backward Pass. We will describe the backward pass only for the particularly efficient truncated gradient version of the LSTM algorithm. For simplicity we will use equal signs even where approximations are made according to the truncated backpropagation equations above. The squared error at time t is given by

)2, (A.14) 其中 \t_k(t)\ 是输出单元 \k\ 在时间 \t\ 的目标。时间 \t\ 对权重 \w_{lm}\ 的基于梯度的更新（学习率为）的贡献为

)2, (A.14) where \t_k(t)\ is output unit \k\'s target at time \t\. Time \t\'s contribution to \w_{lm}\'s gradient-based update with learning rate

(A.15) 我们定义某一单元 \l\ 在时间步 \t\ 的误差为

(A.15) We define some unit \l\'s error at time step \t\ by

\e_l(t) := -\frac{\partial E(t)}{\partial \text{net}_l(t)}\ . (A.16) 使用（几乎）标准的反向传播，我们首先计算到输出单元（\l = k\）、隐藏单元（\l = i\）和输出门（\l = \text{out}_j\）的权重的更新。我们得到（比较公式 A.8、A.9 和 A.11）：

\e_l(t) := -\frac{\partial E(t)}{\partial \text{net}_l(t)}\. (A.16) Using (almost) standard backpropagation, we first compute updates for weights to output units (\l = k\), weights to hidden units (\l = i\) and weights to output gates (\l = \text{out}_j\). We obtain (compare formulas A.8, A.9, and A.11):

(A.19) 对于所有可能的 \l\，时间 \t\ 对权重 \w_{lm}\ 更新的贡献为

(A.19) For all possible \l\ time \t\'s contribution to \w_{lm}\'s update is

\\Delta w_{lm}(t) = \alpha e_l(t) y_m(t-1)\ . (A.20) 剩余的对输入门（\l = \text{in}_j\）和细胞单元（\l = c_j^v\）权重的更新不太常规。我们定义某一内部状态 \s_{c_j^v}\ 的误差：

\\Delta w_{lm}(t) = \alpha e_l(t) y_m(t-1)\. (A.20) The remaining updates for weights to input gates (\l = \text{in}_j\) and to cell units (\l = c_j^v\) are less conventional. We define some internal state \s_{c_j^v}\'s error:

\(w_{kc vj e_k}(t)\)。 (A.21) 当 l = in_j 或 l = cv_j, v = 1, ..., S_j 时，我们得到

w_{kc vj e_k}(t). (A.21) We obtain for l = in_j or l = cv_j, v = 1, ..., S_j

(A.22) 内部状态对权重的导数以及相应的权重更新如下（比较表达式 A.12）：

(A.22) The derivatives of the internal states with respect to weights and the corresponding weight updates are as follows (compare expression A.12):

\(net_{in_j}(t)\) \(y_m(t-1)\)； (A.23) 因此，时间 t 对 \(w_{in_j m}\) 更新的贡献是（比较表达式 A.8）：

(net_{in_j}(t)) y_m(t-1); (A.23) therefore, time t's contribution to w_{in_j m}'s update is (compare expression A.8):

(A.24) 类似地，我们得到（比较表达式 A.12）：

(A.24) Similarly we get (compare expression A.12):

+ \(g'(net_{cv_j}(t))\) \(f_{in_j}(net_{in_j}(t))\) \(y_m(t-1)\)； (A.25) 因此时间 t 对 \(w_{cv_j m}\) 更新的贡献是（比较表达式 A.8）：

+ g'(net_{cv_j}(t)) f_{in_j}(net_{in_j}(t)) y_m(t-1); (A.25) therefore time t's contribution to w_{cv_j m}'s update is (compare expression A.8):

我们实现反向传播所需的所有公式是方程 A.17 到 A.21 以及 A.23 到 A.26。每个权重的总更新是所有时间步贡献的总和。

All we need to implement for the backward pass are equations A.17 through A.21 and A.23 through A.26. Each weight's total update is the sum of the contributions of all time steps.

A.1.4 计算复杂度。LSTM 每个时间步的更新复杂度为

A.1.4 Computational Complexity. LSTM's update complexity per time step is

O(KH + KCS + HI + CSI) = O(W), (A.27) 其中 K 是输出单元数，C 是记忆单元块数，S > 0 是记忆单元块的大小，H 是隐藏单元数，I 是前向连接到记忆单元、门单元和隐藏单元的（最大）单元数，以及

O(KH + KCS + HI + CSI) = O(W), (A.27) where K is the number of output units, C is the number of memory cell blocks, S > 0 is the size of the memory cell blocks, H is the number of hidden units, I is the (maximal) number of units forward-connected to memory cells, gate units and hidden units, and

W = KH + KCS + CSI + 2CI + HI = O(KH + KCS + CSI + HI)。

W = KH + KCS + CSI + 2CI + HI = O(KH + KCS + CSI + HI).

是权重的数量。表达式 A.27 是通过考虑反向传播的所有计算得到的：方程 A.17 需要 K 步；A.18 需要

is the number of weights. Expression A.27 is obtained by considering all computations of the backward pass: equation A.17 needs K steps; A.18 needs

KH 步；A.19 需要 KSC 步；A.20 需要 K(H+C)步用于输出单元，

KH steps; A.19 needs KSC steps; A.20 needs K(H+C) steps for output units,

HI 步用于隐藏单元，CI 步用于输出门；A.21 需要 KCS 步；A.23 需要 CSI 步；A.24 需要 CSI 步；A.25 需要 CSI 步；A.26 需要

HI steps for hidden units, CI steps for output gates; A.21 needs KCS steps; A.23 needs CSI steps; A.24 needs CSI steps; A.25 needs CSI steps; A.26 needs

CSI 步。总步数为 K + 2KH + KC + 2KSC + HI + CI + 4CSI 步，即

CSI steps. The total is K + 2KH + KC + 2KSC + HI + CI + 4CSI steps, or

O(KH + KSC + HI + CSI)步。我们得出结论：LSTM 算法每时间步的更新复杂度与完全递归网络的 BPTT 类似。在给定时间步，只需存储方程 A.23 和 A.25 中最新的 2 个 CSI 的\∂s_{cvj}/∂w_{lm}\值。因此，LSTM 的存储复杂度也是 O(W)，与输入序列长度无关。

O(KH + KSC + HI + CSI) steps. We conclude that LSTM algorithm’s update complexity per time step is just like BPTT’s for a fully recurrent net. At a given time step, only the 2 CSI most recent \∂s_{cvj}/∂w_{lm}\ values from equations A.23 and A.25 need to be stored. Hence LSTM’s storage complexity also is O(W); it does not depend on the input sequence length.

A.2 误差流。我们计算误差信号在通过记忆单元反向传播 q 个时间步时的缩放程度。作为副产品，该分析再次证实，只要截断反向传播切断试图离开记忆单元的误差流，记忆单元 CEC 内的误差流确实是恒定的（另见第 3.2 节）。该分析还强调了\s_{cj}\可能出现的不期望的长期漂移，以及负偏置输入门的有益抵消作用。使用截断反向传播学习规则，我们得到

A.2 Error Flow. We compute how much an error signal is scaled while flowing back through a memory cell for q time steps. As a by-product, this analysis reconfirms that the error flow within a memory cell’s CEC is indeed constant, provided that truncated backpropagation cuts off error flow trying to leave memory cells (see also section 3.2). The analysis also highlights a potential for undesirable long-term drifts of \s_{cj}\, as well as the beneficial, countermanding influence of negatively biased input gates. Using the truncated backpropagation learning rule, we obtain

\(∂s_{c_j}(t-k) / ∂s_{c_j}(t-k-1) = 1 + (∂y_{in_j}(t-k) / ∂s_{c_j}(t-k-1)) g(net_{c_j}(t-k))\)

∂s_{c_j}(t-k) / ∂s_{c_j}(t-k-1) = 1 + (∂y_{in_j}(t-k) / ∂s_{c_j}(t-k-1)) g(net_{c_j}(t-k))

\+ y_{in_j}(t-k) g'(net_{c_j}(t-k)) (∂net_{c_j}(t-k) / ∂s_{c_j}(t-k-1))\

+ y_{in_j}(t-k) g'(net_{c_j}(t-k)) (∂net_{c_j}(t-k) / ∂s_{c_j}(t-k-1))

\(∂y_{in_j}(t-k) / ∂y_u(t-k-1)) (∂y_u(t-k-1) / ∂s_{c_j}(t-k-1))\

(∂y_{in_j}(t-k) / ∂y_u(t-k-1)) (∂y_u(t-k-1) / ∂s_{c_j}(t-k-1))

\[∂net_{c_j}(t-k) / ∂y_u(t-k-1)] [∂y_u(t-k-1) / ∂s_{c_j}(t-k-1)]\

[∂net_{c_j}(t-k) / ∂y_u(t-k-1)] [∂y_u(t-k-1) / ∂s_{c_j}(t-k-1)]

\≈_{tr}\ 1. (A.28) \≈_{tr}\ 符号表示等式成立，因为截断反向传播将以下导数置为零：

≈_{tr} 1. (A.28) The ≈_{tr} sign indicates equality due to the fact that truncated backpropagation replaces by zero the following derivatives:

\frac{\partial y_{in}^j(t-k)}{\partial y_u(t-k-1)}\ ∀u 和 \frac{\partial net_{c_j}(t-k)}{\partial y_u(t-k-1)}\ ∀u。

\frac{\partial y_{in}^j(t-k)}{\partial y_u(t-k-1)}\ ∀u and \frac{\partial net_{c_j}(t-k)}{\partial y_u(t-k-1)}\ ∀u.

接下来，一个误差 ϑ_j(t) 开始在 c_j 的输出处反向传播。我们重新定义

In what follows, an error ϑ_j(t) starts flowing back at c_j's output. We re-define

w_{ic_j} ϑ_i(t+1) 。(A.29) 根据第 3.1 节的定义和约定，我们计算截断反向传播学习规则的误差流。输出门处的误差为

w_{ic_j} ϑ_i(t+1) . (A.29) Following the definitions and conventions of section 3.1, we compute error flow for the truncated backpropagation learning rule. The error occurring at the output gate is

\frac{\partial y_{out}^j(t)}{\partial net_{out}^j(t)}\ \frac{\partial y_{c_j}(t)}{\partial y_{out}^j(t)}\ ϑ_j(t) 。(A.30) 内部状态处的误差为

\frac{\partial y_{out}^j(t)}{\partial net_{out}^j(t)}\ \frac{\partial y_{c_j}(t)}{\partial y_{out}^j(t)}\ ϑ_j(t) . (A.30) The error occurring at the internal state is

ϑ_{s_{c_j}}(t) = \frac{\partial s_{c_j}(t+1)}{\partial s_{c_j}(t)}\ ϑ_{s_{c_j}}(t+1) + \frac{\partial y_{c_j}(t)}{\partial s_{c_j}(t)}\ ϑ_j(t) 。(A.31) 由于我们使用截断反向传播，我们有

ϑ_{s_{c_j}}(t) = \frac{\partial s_{c_j}(t+1)}{\partial s_{c_j}(t)}\ ϑ_{s_{c_j}}(t+1) + \frac{\partial y_{c_j}(t)}{\partial s_{c_j}(t)}\ ϑ_j(t) . (A.31) Since we use truncated backpropagation we have

其中 θ_i(t+1)；因此，Sepp Hochreiter 和 Jürgen Schmidhuber 得到

where θ_i(t+1); Sepp Hochreiter and Jürgen Schmidhuber therefore we get

∂θ_i(t+1)/∂θ_{s c_j}(t+1) ≈ 0。 (A.32) 方程 (A.31) 和 (A.32) 表明通过记忆单元内部状态的误差流恒定：

\partial θ_i(t+1) / \partial θ_{s c_j}(t+1) ≈ 0. (A.32) Equations (A.31) and (A.32) imply constant error flow through internal states of memory cells:

∂θ_{s c_j}(t)/∂θ_{s c_j}(t+1) = ∂s_{c_j}(t+1)/∂s_{c_j}(t) ≈ 1。 (A.33) 发生在记忆单元输入处的误差为

\partial θ_{s c_j}(t) / \partial θ_{s c_j}(t+1) = \partial s_{c_j}(t+1) / \partial s_{c_j}(t) ≈ 1. (A.33) The error occurring at the memory cell input is

θ_{c_j}(t) = ∂g(net_{c_j}(t))/∂net_{c_j}(t) * ∂s_{c_j}(t)/∂g(net_{c_j}(t)) * θ_{s c_j}(t)。 (A.34) 发生在输入门处的误差为

\theta_{c_j}(t) = \frac{\partial g(\text{net}_{c_j}(t))}{\partial \text{net}_{c_j}(t)} \frac{\partial s_{c_j}(t)}{\partial g(\text{net}_{c_j}(t))} \theta_{s c_j}(t). (A.34) The error occurring at the input gate is

∂y_{in_j}(t)/∂net_{in_j}(t) * ∂s_{c_j}(t)/∂y_{in_j}(t) * θ_{s c_j}(t)。 (A.35)

\partial y_{in_j}(t) / \partial \text{net}_{in_j}(t) \partial s_{c_j}(t) / \partial y_{in_j}(t) \theta_{s c_j}(t). (A.35)

A.2.1 无外部误差流。误差沿带有权重 \(w_{lv}\) 的输出连接从单元 l 反向传播到单元 v。此时，“外部误差”（注意对于常规单元而言，只有外部误差）

A.2.1 No External Error Flow. Errors are propagated back from units l to unit v along outgoing connections with weights \(w_{lv}\). This “external error” (note that for conventional units there is nothing but external error) at time

\(\partial net_l(t+1) \partial y_v(t) \vartheta_l(t+1)\) 。(A.36) 我们得到

\(\partial net_l(t+1) \partial y_v(t) \vartheta_l(t+1)\). (A.36) We obtain

\(\partial \vartheta_{ev}(t-1) \partial \vartheta_j(t) = \partial y_v(t-1) \partial net_v(t-1)\)

\(\partial \vartheta_{ev}(t-1) \partial \vartheta_j(t) = \partial y_v(t-1) \partial net_v(t-1)\)

\(\partial \vartheta_{out_j}(t) \partial \vartheta_j(t) \partial net_{out_j}(t) \partial y_v(t-1)\)

\(\partial \vartheta_{out_j}(t) \partial \vartheta_j(t) \partial net_{out_j}(t) \partial y_v(t-1)\)

\(+ \partial \vartheta_{in_j}(t) \partial \vartheta_j(t) \partial net_{in_j}(t) \partial y_v(t-1) + \partial \vartheta_{c_j}(t) \partial \vartheta_j(t) \partial net_{c_j}(t) \partial y_v(t-1)\)

\(+ \partial \vartheta_{in_j}(t) \partial \vartheta_j(t) \partial net_{in_j}(t) \partial y_v(t-1) + \partial \vartheta_{c_j}(t) \partial \vartheta_j(t) \partial net_{c_j}(t) \partial y_v(t-1)\)

(A.37) 我们观察到，到达记忆单元输出端的误差θ_j 不会通过与 in_j、out_j、c_j 的外部连接反向传播到单元 v。

(A.37) We observe that the error θ_j arriving at the memory cell output is not back-propagated to units v by external connections to in_j, out_j, c_j.

A.2.2 记忆单元内部的误差流。我们现在关注记忆单元 CEC 内部的误差反向流动。这实际上是唯一能够跨越多个时间步的误差流。假设误差θ_j(t)在时间 t 到达 c_j 的输出，并向后传播 q 步，直到到达 in_j 或记忆单元输入 g(net_{c_j})。它被一个因子缩放：

A.2.2 Error Flow Within Memory Cells. We now focus on the error back-flow within a memory cell's CEC. This is actually the only type of error flow that can bridge several time steps. Suppose error θ_j(t) arrives at c_j's output at time t and is propagated back for q steps until it reaches in_j or the memory cell input g(net_{c_j}). It is scaled by a factor of

q > 0。 (A.38) 展开方程 A.38，我们得到

q > 0. (A.38) Expanding equation A.38, we obtain

\frac{\partial \vartheta_v(t-q)}{\partial \vartheta_{s_{c_j}}(t-q)} \frac{\partial \vartheta_{s_{c_j}}(t-q)}{\partial \vartheta_j(t)} = g'(net_{c_j}(t-q)) y^{in_j}(t-q) v = c_j g(net_{c_j}(t-q)) f'

\frac{\partial \vartheta_v(t-q)}{\partial \vartheta_{s_{c_j}}(t-q)} \frac{\partial \vartheta_{s_{c_j}}(t-q)}{\partial \vartheta_j(t)} = g'(net_{c_j}(t-q)) y^{in_j}(t-q) v = c_j g(net_{c_j}(t-q)) f'

≈tr 0 . (A.37) We observe that the error ϑj arriving at the memory cell output is not back-propagated to units v by external connections to in j, out j, cj.

考虑上一个方程最后一个表达式中的因子。显然，误差流仅在时刻 t（当它进入细胞时）和 t - q 时被缩放。

\(net_{in_j}(t - q)) v = in_j\. (A.39) Consider the factors in the previous equation's last expression. Obviously, error flow is scaled only at times t (when it enters the cell) and t - q

（当它离开细胞时），但中间不被缩放（通过 CEC 的恒定误差流）。我们观察到：1. 输出门的效果是 \yout_j(t)\ 会缩小那些可以在训练早期不使用记忆细胞就能减少的误差。它也会缩小由后期训练阶段使用（激活/停用）记忆细胞导致的误差。如果没有输出门，记忆细胞可能会在看似已经可控的情况下突然开始引起可避免的误差（因为不使用记忆细胞时容易减少相应误差）。参见第 3 节中的“输出权重冲突”和第 4.7 节中的“滥用问题与解决方案”。2. 如果 \sc_j(t)\ 有大的正值或负值（因为 \sc_j\ 自从时间步 \t - q\ 以来发生了漂移），那么 \h'(sc_j(t))\ 可能很小（假设 \h\ 是逻辑 Sigmoid 函数）。参见第 4 节。记忆细胞内部状态 \sc_j\ 的漂移可以通过对输入门 \in_j\ 施加负偏置来抵消（参见第 4 节和下一点）。回忆第 4 节，精确的偏置值并不太重要。3. \y_in_j(t - q)\ 和 \f'(net_{in_j}(t - q))\

(when it leaves the cell), but not in between (constant error flow through the CEC). We observe: 1. The output gate's effect is \yout_j(t)\ scales down those errors that can be reduced early during training without using the memory cell. It also scales down those errors resulting from using (activating/deactivating) the memory cell at later training stages. Without the output gate, the memory cell might, for instance, suddenly start causing avoidable errors in situations that already seemed under control (because it was easy to reduce the corresponding errors without memory cells). See "Output Weight Conflict" in section 3 and "Abuse Problem and Solution" (section 4.7). 2. If there are large positive or negative \sc_j(t)\ values (because \sc_j\ has drifted since time step \t - q\), then \h'(sc_j(t))\ may be small (assuming that \h\ is a logistic sigmoid). See section 4. Drifts of the memory cell's internal state \sc_j\ can be countermanded by negatively biasing the input gate \in_j\ (see section 4 and the next point). Recall from section 4 that the precise bias value does not matter much. 3. \y_in_j(t - q)\ and \f'(net_{in_j}(t - q))\

在输入门被负偏置时很小（假设 \f_{in_j}\ 是逻辑 Sigmoid）。然而，与内部状态 \sc_j\ 漂移的潜在重要性相比，这一点的潜在重要性可以忽略。上述某些因子可能会缩小 LSTM 的整体误差流，但这不是以依赖于时间滞后长度的方式。该流仍然比没有记忆细胞时指数级（阶数为 q）衰减的流有效得多。

are small if the input gate is negatively biased (assume \f_{in_j}\ is a logistic sigmoid). However, the potential significance of this is negligible compared to the potential significance of drifts of the internal state \sc_j\. Some of the factors above may scale down LSTM's overall error flow, but not in a manner that depends on the length of the time lag. The flow will still be much more effective than an exponentially (of order q) decaying flow without memory cells.

感谢 Mike Mozer、Wilfried Brauer、Nic Schraudolph 以及几位匿名审稿人的宝贵评论和建议，它们帮助改进了本文的先前版本（Hochreiter and Schmidhuber, 1995）。本工作得到了德国研究基金会 DFG grant SCHM 942/3-1 的支持。 Almeida, L. B. (1987). A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. In IEEE 1st International Conference on Neural Networks, San Diego (Vol. 2, pp. 609–618). Baldi, P., & Pineda, F. (1991). Contrastive learning and neural oscillator. Neural Computation, 3, 526–545. Bengio, Y., & Frasconi, P. (1994). Credit assignment through time: Alternatives to backpropagation. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems 6 (pp. 75–82). San Mateo, CA: Morgan Kaufmann. Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166. Cleeremans, A., Servan-Schreiber, D., & McClelland, J. L. (1989). Finite-state automata and simple recurrent networks. Neural Computation, 1, 372–381. de Vries, B., & Principe, J. C. (1991). A theory for neural networks with time delays. In R. P. Lippmann, J. E. Moody, & D. S. Touretzky (Eds.), Advances in neural information processing systems 3, (pp. 162–168). San Mateo, CA: Morgan Kaufmann. Doya, K. (1992). Bifurcations in the learning of recurrent neural networks. In Proceedings of 1992 IEEE International Symposium on Circuits and Systems

Thanks to Mike Mozer, Wilfried Brauer, Nic Schraudolph, and several anonymous referees for valuable comments and suggestions that helped to improve a previous version of this article (Hochreiter and Schmidhuber, 1995). This work was supported by DFG grant SCHM 942/3-1 from Deutsche Forschungsgemeinschaft. Almeida, L. B. (1987). A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. In IEEE 1st International Conference on Neural Networks, San Diego (Vol. 2, pp. 609–618). Baldi, P., & Pineda, F. (1991). Contrastive learning and neural oscillator. Neural Computation, 3, 526–545. Bengio, Y., & Frasconi, P. (1994). Credit assignment through time: Alternatives to backpropagation. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems 6 (pp. 75–82). San Mateo, CA: Morgan Kaufmann. Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166. Cleeremans, A., Servan-Schreiber, D., & McClelland, J. L. (1989). Finite-state automata and simple recurrent networks. Neural Computation, 1, 372–381. de Vries, B., & Principe, J. C. (1991). A theory for neural networks with time delays. In R. P. Lippmann, J. E. Moody, & D. S. Touretzky (Eds.), Advances in neural information processing systems 3, (pp. 162–168). San Mateo, CA: Morgan Kaufmann. Doya, K. (1992). Bifurcations in the learning of recurrent neural networks. In Proceedings of 1992 IEEE International Symposium on Circuits and Systems

(net in j (t − q)) v = in j .(A.39) Consider the factors in the previous equation’s last expression. Obvi-ously, error flow is scaled only at times t (when it enters the cell) and t − q

(pp. 65–72). Amsterdam: IOS Press. Hochreiter, S., & Schmidhuber, J. (1997). LSTM can solve hard long time lag problems. In Advances in neural information processing systems 9. Cambridge, MA: MIT Press. Lang, K., Waibel, A., & Hinton, G. E. (1990). A time-delay neural network architecture for isolated word recognition. Neural Networks, 3, 23–43. Lin, T., Horne, B. G., Tino, P., & Giles, C. L. (1996). Learning long-term dependencies in NARX recurrent neural networks. IEEE Transactions on Neural Networks, 7, 1329–1338. Miller, C. B., & Giles, C. L. (1993). Experimental comparison of the effect of order in recurrent neural networks. International Journal of Pattern Recognition and Artificial Intelligence, 7(4), 849–872. Mozer, M. C. (1989). A focused back-propagation algorithm for temporal sequence recognition. Complex Systems, 3, 349–381. Mozer, M. C. (1992). Induction of multiscale temporal structure. In J. E. Moody, S. J. Hanson, & R. P. Lippman (Eds.), Advances in neural information processing systems 4 (pp. 275–282). San Mateo, CA: Morgan Kaufmann. Pearlmutter, B. A. (1989). Learning state space trajectories in recurrent neural networks. Neural Computation, 1(2), 263–269. Pearlmutter, B. A. (1995). Gradient calculations for dynamic recurrent neural networks: A survey. IEEE Transactions on Neural Networks, 6(5), 1212–1228. Pineda, F. J. (1987). Generalization of back-propagation to recurrent neural networks. Physical Review Letters, 19(59), 2229–2232. Pineda, F. J. (1988). Dynamics and architecture for neural computation. Journal of Complexity, 4, 216–245. Plate, T. A. (1993). Holographic recurrent networks. In S. J. Hanson, J. D. Cowan, & C. L. Giles (Eds.), Advances in neural information processing systems 5 (pp. 34–41). San Mateo, CA: Morgan Kaufmann. Pollack, J. B. (1991). Language induction by phase transition in dynamical recognizers. In R. P. Lippmann, J. E. Moody, & D. S. Touretzky (Eds.), Advances in neural information processing systems 3 (pp. 619–626). San Mateo, CA: Morgan Kaufmann. Puskorius, G. V., and Feldkamp, L. A. (1994). Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks. IEEE Transactions on Neural Networks, 5(2), 279–297. Ring, M. B. (1993). Learning sequential tasks by incrementally adding higher orders. In S. J. Hanson, J. D. Cowan, & C. L. Giles (Eds.), Advances in neural information processing systems 5 (pp. 115–122). San Mateo, CA: Morgan Kaufmann. 1780 Sepp Hochreiter and Jürgen Schmidhuber Robinson, A. J., & Fallside, F. (1987). The utility driven dynamic error propagation network (Tech. Rep. No. CUED/F-INFENG/TR.1). Cambridge: Cambridge University Engineering Department. Schmidhuber, J. (1989). A local learning algorithm for dynamic feedforward and recurrent networks. Connection Science, 1(4), 403–412. Schmidhuber, J. (1992a). A fixed size storage O(n3) time complexity learning algorithm for fully recurrent continually running networks. Neural Computation, 4(2), 243–248. Schmidhuber, J. (1992b). Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2), 234–242. Schmidhuber, J. (1992c). Learning unambiguous reduced sequence descriptions. In J. E. Moody, S. J. Hanson, & R. P. Lippman (Eds.), Advances in neural information processing systems 4 (pp. 291–298). San Mateo, CA: Morgan Kaufmann. Schmidhuber, J. (1993). Netzwerkarchitekturen, Zielfunktionen und Kettenregel. Habilitationsschrift, Institut für Informatik, Technische Universität München. Schmidhuber, J., & Hochreiter, S. (1996). Guessing can outperform many long time lag algorithms (Tech. Rep. No. IDSIA-19-96). Lugano, Switzerland: Instituto Dalle Molle di Studi sull’Intelligenza Artificiale. Silva, G. X., Amaral, J. D., Langlois, T., & Almeida, L. B. (1996). Faster training of recurrent networks. In F. L. Silva, J. C. Principe, & L. B. Almeida (Eds.), Spatiotemporal models in biological and artificial systems (pp. 168–175). Amsterdam: IOS Press. Smith, A. W., & Zipser, D. (1989). Learning sequential structures with the real-time recurrent learning algorithm. International Journal of Neural Systems, 1(2), 125–131. Sun, G., Chen, H., & Lee, Y. (1993). Time warping invariant neural networks. In S. J. Hanson, J. D. Cowan, & C. L. Giles (Eds.), Advances in neural information processing systems 5 (pp. 180–187). San Mateo, CA: Morgan Kaufmann. Watrous, R. L., & Kuhn, G. M. (1992). Induction of finite-state languages using second-order recurrent networks. Neural Computation, 4, 406–414. Werbos, P. J. (1988). Generalization of backpropagation with application to a recurrent gas market model. Neural Networks, 1, 339–356. Williams, R. J. (1989). Complexity of exact gradient computation algorithms for recurrent neural networks (Tech. Rep. No. NU-CCS-89-27). Boston: Northeastern University, College of Computer Science. Williams, R. J. & Peng, J. (1990). An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural Computation, 4, 491–501. Williams, R. J., & Zipser, D. (1992). Gradient-based learning algorithms for recurrent networks and their computational complexity. In Y. Chauvin, & D. E. Rumelhart (Eds.), Back-propagation: Theory, architectures and applications. Hillsdale, NJ: Erlbaum. Received August 28, 1995; accepted February 24, 1997.

(pp. 65–72). Amsterdam: IOS Press. Hochreiter, S., & Schmidhuber, J. (1997). LSTM can solve hard long time lag problems. In Advances in neural information processing systems 9. Cambridge, MA: MIT Press. Lang, K., Waibel, A., & Hinton, G. E. (1990). A time-delay neural network architecture for isolated word recognition. Neural Networks, 3, 23–43. Lin, T., Horne, B. G., Tino, P., & Giles, C. L. (1996). Learning long-term dependencies in NARX recurrent neural networks. IEEE Transactions on Neural Networks, 7, 1329–1338. Miller, C. B., & Giles, C. L. (1993). Experimental comparison of the effect of order in recurrent neural networks. International Journal of Pattern Recognition and Artificial Intelligence, 7(4), 849–872. Mozer, M. C. (1989). A focused back-propagation algorithm for temporal sequence recognition. Complex Systems, 3, 349–381. Mozer, M. C. (1992). Induction of multiscale temporal structure. In J. E. Moody, S. J. Hanson, & R. P. Lippman (Eds.), Advances in neural information processing systems 4 (pp. 275–282). San Mateo, CA: Morgan Kaufmann. Pearlmutter, B. A. (1989). Learning state space trajectories in recurrent neural networks. Neural Computation, 1(2), 263–269. Pearlmutter, B. A. (1995). Gradient calculations for dynamic recurrent neural networks: A survey. IEEE Transactions on Neural Networks, 6(5), 1212–1228. Pineda, F. J. (1987). Generalization of back-propagation to recurrent neural networks. Physical Review Letters, 19(59), 2229–2232. Pineda, F. J. (1988). Dynamics and architecture for neural computation. Journal of Complexity, 4, 216–245. Plate, T. A. (1993). Holographic recurrent networks. In S. J. Hanson, J. D. Cowan, & C. L. Giles (Eds.), Advances in neural information processing systems 5 (pp. 34–41). San Mateo, CA: Morgan Kaufmann. Pollack, J. B. (1991). Language induction by phase transition in dynamical recognizers. In R. P. Lippmann, J. E. Moody, & D. S. Touretzky (Eds.), Advances in neural information processing systems 3 (pp. 619–626). San Mateo, CA: Morgan Kaufmann. Puskorius, G. V., and Feldkamp, L. A. (1994). Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks. IEEE Transactions on Neural Networks, 5(2), 279–297. Ring, M. B. (1993). Learning sequential tasks by incrementally adding higher orders. In S. J. Hanson, J. D. Cowan, & C. L. Giles (Eds.), Advances in neural information processing systems 5 (pp. 115–122). San Mateo, CA: Morgan Kaufmann. 1780 Sepp Hochreiter and Jürgen Schmidhuber Robinson, A. J., & Fallside, F. (1987). The utility driven dynamic error propagation network (Tech. Rep. No. CUED/F-INFENG/TR.1). Cambridge: Cambridge University Engineering Department. Schmidhuber, J. (1989). A local learning algorithm for dynamic feedforward and recurrent networks. Connection Science, 1(4), 403–412. Schmidhuber, J. (1992a). A fixed size storage O(n3) time complexity learning algorithm for fully recurrent continually running networks. Neural Computation, 4(2), 243–248. Schmidhuber, J. (1992b). Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2), 234–242. Schmidhuber, J. (1992c). Learning unambiguous reduced sequence descriptions. In J. E. Moody, S. J. Hanson, & R. P. Lippman (Eds.), Advances in neural information processing systems 4 (pp. 291–298). San Mateo, CA: Morgan Kaufmann. Schmidhuber, J. (1993). Netzwerkarchitekturen, Zielfunktionen und Kettenregel. Habilitationsschrift, Institut für Informatik, Technische Universität München. Schmidhuber, J., & Hochreiter, S. (1996). Guessing can outperform many long time lag algorithms (Tech. Rep. No. IDSIA-19-96). Lugano, Switzerland: Instituto Dalle Molle di Studi sull’Intelligenza Artificiale. Silva, G. X., Amaral, J. D., Langlois, T., & Almeida, L. B. (1996). Faster training of recurrent networks. In F. L. Silva, J. C. Principe, & L. B. Almeida (Eds.), Spatiotemporal models in biological and artificial systems (pp. 168–175). Amsterdam: IOS Press. Smith, A. W., & Zipser, D. (1989). Learning sequential structures with the real-time recurrent learning algorithm. International Journal of Neural Systems, 1(2), 125–131. Sun, G., Chen, H., & Lee, Y. (1993). Time warping invariant neural networks. In S. J. Hanson, J. D. Cowan, & C. L. Giles (Eds.), Advances in neural information processing systems 5 (pp. 180–187). San Mateo, CA: Morgan Kaufmann. Watrous, R. L., & Kuhn, G. M. (1992). Induction of finite-state languages using second-order recurrent networks. Neural Computation, 4, 406–414. Werbos, P. J. (1988). Generalization of backpropagation with application to a recurrent gas market model. Neural Networks, 1, 339–356. Williams, R. J. (1989). Complexity of exact gradient computation algorithms for recurrent neural networks (Tech. Rep. No. NU-CCS-89-27). Boston: Northeastern University, College of Computer Science. Williams, R. J. & Peng, J. (1990). An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural Computation, 4, 491–501. Williams, R. J., & Zipser, D. (1992). Gradient-based learning algorithms for recurrent networks and their computational complexity. In Y. Chauvin, & D. E. Rumelhart (Eds.), Back-propagation: Theory, architectures and applications. Hillsdale, NJ: Erlbaum. Received August 28, 1995; accepted February 24, 1997.