Learning Internal Representations by Error Propagation
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→通过误差传播学习内部表示 D. E. RUMELHART、G. E. HINTON 和 R. J. WILLIAMS 问题 我们现在对简单的两层关联网络有了相当好的理解，其中一组到达输入层的输入模式直接映射到输出层的一组输出模式。这种网络没有隐藏单元，仅涉及输入和输出
Learning Internal Representations by Error Propagation D. E. RUMELHART, G. E. HINTON, and R. J. WILLIAMS THE PROBLEM We now have a rather good understanding of simple two-layer associative networks in which a set of input patterns arriving at an input layer are mapped directly to a set of output patterns at an output layer. Such networks have no hidden units. They involve only input and output
通过误差传播学习内部表示
Learning Internal Representations by Error Propagation
D. E. RUMELHART、G. E. HINTON 和 R. J. WILLIAMS 问题
D. E. RUMELHART, G. E. HINTON, and R. J. WILLIAMS THE PROBLEM
我们现在对简单的两层关联网络有了相当好的理解，其中一组到达输入层的输入模式直接映射到输出层的一组输出模式。这种网络没有隐藏单元，仅涉及输入和输出
We now have a rather good understanding of simple two-layer associative networks in which a set of input patterns arriving at an input layer are mapped directly to a set of output patterns at an output layer. Such networks have no hidden units. They involve only input and output
单元。在这些情况下，不存在内部表示。外部世界提供的编码必须足够。这些网络已被证明
units. In these cases there is no internal representation. The coding provided by the external world must suffice. These networks have proved
在广泛的应用中非常有用（参见第 2、17 和 18 章）。
useful in a wide variety of applications (cf. Chapters 2, 17, and 18).
或许这类网络的基本特征是将相似的输入模式映射到相似的输出模式。
Perhaps the essential character of such networks is that they map similar input patterns to similar output patterns.
PDP 系统中模式的相似性由其重叠程度决定。这类网络中的重叠是在学习系统之外由产生模式的任何因素决定的。
The similarity of patterns in a PDP system is determined by their overlap. The overlap in such networks is determined outside the learning system itself—by whatever produces the patterns.
相似输入模式导致相似输出的约束可能导致系统无法学习某些输入到输出的映射。当外部世界提供的表示使得输入和输出模式的相似性结构非常不同时，没有隐藏单元（即两层网络）的网络将无法学习所需的映射。
The constraint that similar input patterns lead to similar outputs can lead to an inability of the system to learn certain mappings from input to output. Whenever the representation provided by the outside world is such that the similarity structure of the input and output patterns are very different, a network without hidden units (i.e., a two-layer network) will be unable to learn the required mappings.
没有隐藏单元的网络将无法执行必要的映射。这种情况的一个经典例子是表 1 所示的异或（XOR）问题。
A network without hidden units will be unable to perform the necessary mappings. A classic example of this case is the exclusive-or (XOR) problem illustrated in Table 1.
这里我们看到，那些重叠最少的模式本应产生相同的输出值。这个问题以及许多类似的问题无法由没有隐藏单元的网络解决，因为隐藏单元能够创建输入模式自身的内部表示。有趣的是，如果输入模式包含一个第三输入，每当前两个输入值为 1 时该第三输入取值为 1，如表 2 所示，那么一个两层系统就能解决该问题。Minsky 和 Papert（1969）对这类系统能够执行所需映射的条件进行了非常仔细的分析。他们指出，在许多有趣的情况下，这类网络无法解决这些问题。另一方面，正如 Minsky 和 Papert 也指出的，如果有一层简单的类似感知器的隐藏单元，如图 1 所示，它可以增强原始输入模式，那么总有一种
Here we see that those patterns which overlap least are supposed to generate identical output values. This problem and many others like it cannot be performed by networks without hidden units with which to create their own internal representations of the input patterns. It is interesting to note that had the input patterns contained a third input taking the value 1 whenever the first two have value 1 as shown in Table 2, a two-layer system would be able to solve the problem. Minsky and Papert (1969) have provided a very careful analysis of conditions under which such systems are capable of carrying out the required mappings. They show that in a large number of interesting cases, networks of this kind are incapable of solving the problems. On the other hand, as Minsky and Papert also pointed out, if there is a layer of simple perceptron-like hidden units, as shown in Figure 1, with which the original input pattern can be augmented, there is always a
输入模式在隐藏单元中的重新编码（即内部表示），其中隐藏单元之间模式的相似性可以支持从输入单元到输出单元的任何必要映射。因此，如果我们有从输入单元到足够大的一组隐藏单元的正确连接，我们总能找到一个通过这些隐藏单元执行从输入到输出的任何映射的表示。对于 XOR 问题，增加一个检测输入单元合取的特征会改变相似性
Recoding (i.e., an internal representation) of the input patterns in the hidden units in which the similarity of the patterns among the hidden units can support any required mapping from the input to the output units. Thus, if we have the right connections from the input units to a large enough set of hidden units, we can always find a representation that will perform any mapping from input to output through these hidden units. In the case of the XOR problem, the addition of a feature that detects the conjunction of the input units changes the similarity
受版权保护的材料 320 基本机制
Copyrighted Material 320 BASIC MECHANISMS
I. 多层网络。在这种情况下，到达输入单元的信息被重新编码为内部表示，输出由中间
I. A multilayer network. In this case the information coming to the input units is recoded into an internal representation and the outputs are generated by the inter-
原始模式。如果有足够多的隐藏单元，输入模式总可以被编码成一种形式，使得适当的输出模式
the original pattern. Input patterns can always be encoded, if there are enough hidden units, in a form so that the appropriate output pat-
模式的结构足以允许学习解决方案。如图 2 所示，这可以通过单个隐藏单元实现。箭头上的数字表示单元之间连接的强度。圆圈内的数字表示单元的阈值。隐藏单元阈值+1.5 确保只有当两个输入单元都激活时它才会开启
structure of the patterns sufficiently to allow the solution to be learned. As illustrated in Figure 2, this can be done with a single hidden unit. The numbers on the arrows represent the strengths of the connections among the units. The numbers written in the circles represent the thresholds of the units. The value of +1.5 for the threshold of the hidden unit ensures that it will be turned on only when both input units
输出单元设值为 0.5，确保只有当接收到大于 0.5 的净正输入时才会开启。从隐藏单元到输出单元的权重-2 确保当两个输入单元都开启时，输出单元不会开启。注意，从输出单元的角度来看，隐藏单元仅被视为另一个输入单元。就好像输出模式由三个而不是两个单元组成。
are on. The value 0.5 for the output unit ensures that it will turn on only when it receives a net positive input greater than 0.5. The weight of -2 from the hidden unit to the output unit ensures that the output unit will not come on when both input units are on. Note that from the point of view of the output unit, the hidden unit is treated as simply another input unit. It is as if the output patterns consisted of three rather than two units.
版权所有 8. 学习内部表示 321
Copyrighted Material 8. LEARNING INTERNAL REPRESENTATIONS 321
诸如这样的网络的存在展示了隐藏单元和内部表示的潜在力量。正如 Minsky 和 Papert 所指出的，问题在于，对于所有可以在没有隐藏单元的情况下解决的问题，存在一个非常简单的有保证的学习规则，即感知机收敛过程（或最初由 Widrow 和 Hoff 在 1960 年提出的变体，我们称之为 delta 规则；见第 11 章），但对于带有隐藏单元的网络，却没有同样强大的学习规则。对此缺乏，主要有三种回应。一种回应是竞争学习（第 5 章），其中采用简单的无监督学习规则来发展有用的隐藏单元。虽然这些方法很有前景，但没有外部力量确保发展出适合所需映射的隐藏单元。第二种回应是简单地
The existence of networks such as this illustrates the potential power of hidden units and internal representations. The problem, as noted by Minsky and Papert, is that whereas there is a very simple guaranteed learning rule for all problems that can be solved without hidden units, namely, the perceptron convergence procedure (or the variation due originally to Widrow and Hoff, 1960, which we call the delta rule; see Chapter 11), there is no equally powerful rule for learning in networks with hidden units. There have been three basic responses to this lack. One response is represented by competitive learning (Chapter 5) in which simple unsupervised learning rules are employed so that useful hidden units develop. Although these approaches are promising, there is no external force to ensure that hidden units appropriate for the required mapping are developed. The second response is to simply
基于某些先验理由假设一个看似合理的内部表示。这是在动词学习章节（第 18 章）和单词感知的交互激活模型中所采取的策略。
assume an internal representation that, on some a priori grounds, seems reasonable. This is the tack taken in the chapter on verb learning (Chapter 18) and in the interactive activation model of word perception
are on. The val ue 0 . 5 for the ou tp ut unit insures that i t will tu rn on only when it receives a net posi tive i n put greater than 0 . 5 . The weight of - 2 from the h idden unit to the output unit insures that the output unit wi l l not come on whe n both i nput units are on. Note that from the po int of view of the out p ut un it, the hidden unit is treated as s imply another input unit . It is as jf the jOP'ut . p'atterns consi sted of three rather than two units .
(McClelland & Rumelhart, 1981; Rumelhart & McClelland, 1982)。
(McClelland & Rumelhart, 1981; Rumelhart & McClelland, 1982).
第三种方法是尝试开发一种能够学习足够执行当前任务的内部表示的学习过程。其中一个这样的发展出现在关于
The third approach is to attempt to develop a learning procedure capable of learning an internal representation adequate for performing the task at hand. One such development is presented in the discussion of
第 7 章中的玻尔兹曼机。正如我们所看到的，这个过程涉及使用随机单元，要求网络在两个不同阶段达到平衡，并且仅限于对称网络。另一个最近的方法，也采用随机单元，由 Barto 及其同事提出（参见 Barto 322 BASIC MECHANISMS
Boltzmann machines in Chapter 7. As we have seen, this procedure involves the use of stochastic units, requires the network to reach equilibrium in two different phases, and is limited to symmetric networks. Another recent approach, also employing stochastic units, has been developed by Barto and his colleagues (cf. Barto 322 BASIC MECHANISMS
& Anandan, 1985）。在本章中，我们提出另一种替代方法，它使用确定性单元，仅涉及局部计算，并且是 delta 规则的清晰推广。我们称之为广义 delta 规则。从其他考虑出发，Parker（1985）独立推导出了类似的推广，他称之为学习逻辑。Le Cun（1985）也研究了大体相似的学习方案。在本章的其余部分，我们首先推导广义 delta 规则，然后通过提供一些模拟结果来说明其使用，最后指出基本思想的一些进一步推广。
& Anandan, 1985). In this chapter we present another alternative that works with deterministic units, that involves only local computations, and that is a clear generalization of the delta rule. We call this the generalized delta rule. From other considerations, Parker (1985) has independently derived a similar generalization, which he calls learning logic. Le Cun (1985) has also studied a roughly similar learning scheme. In the remainder of this chapter we first derive the generalized delta rule, then we illustrate its use by providing some results of our simulations, and finally we indicate some further generalizations of the basic idea.
我们提出的学习过程涉及呈现一组输入和输出模式对。系统首先使用输入向量生成自己的输出向量，然后将其与期望输出（或目标向量）进行比较。如果没有差异，则不进行学习。否则，改变权重以减少差异。在这种情况下，如果没有隐藏单元，这将生成标准的 delta 规则，如第 2 章和第 11 章所述。在呈现输入/输出对 p 之后改变权重的规则由下式给出
The learning procedure we propose involves the presentation of a set of pairs of input and output patterns. The system first uses the input vector to produce its own output vector and then compares this with the desired output, or target vector. If there is no difference, no learning takes place. Otherwise the weights are changed to reduce the difference. In this case, with no hidden units, this generates the standard delta rule as described in Chapters 2 and 11. The rule for changing weights following presentation of input/output pair p is given by
其中 \t_{pj}⟑ 是模式 p 的输出模式第 j 个分量的目标输入，\o_{pj}⟑ 是输入模式 p 呈现后产生的实际输出模式的第 j 个元素，\i_{pi}⟑ 是输入模式第 i 个元素的值，\δ_{pj} = t_{pj} - o_{pj}⟑，而 \Δ_p w_{ji}⟑ 是在模式 p 呈现后对从第 i 个单元到第 j 个单元的权重所做的改变。
where t_{pj} is the target input for the j-th component of the output pattern for pattern p, o_{pj} is the j-th element of the actual output pattern produced by the presentation of input pattern p, i_{pi} is the value of the i-th element of the input pattern, \δ_{pj} = t_{pj} - o_{pj}⟑, and \Δ_p w_{ji}⟑ is the change to be made to the weight from the i-th to the j-th unit following presentation of pattern p.
有许多推导该规则的方法。就当前目的而言，值得看到对于线性单元，它最小化了实际输出值与期望输出值之差的平方，这些差值是在输出单元以及所有输入/输出向量对上求和得到的。证明这一点的一种方式是，表明误差度量对每个权重的导数与 delta 规则所规定的权重变化成正比，且比例常数为负。这相当于在权重空间中进行最速下降，其中权重空间中任意一点的高度等于误差度量。（注意以下部分
There are many ways of deriving this rule. For present purposes, it is useful to see that for linear units it minimizes the squares of the differences between the actual and the desired output values summed over the output units and all pairs of input/output vectors. One way to show this is to show that the derivative of the error measure with respect to each weight is proportional to the weight change dictated by the delta rule, with negative constant of proportionality. This corresponds to performing steepest descent on a surface in weight space whose height at any point in weight space is equal to the error measure. (Note that some of the following sections
版权材料 8. 学习内部表示 323
Copyrighted Material 8. LEARNING INTERNAL REPRESENTATIONS 323
以斜体书写。这些部分构成了对周围文本中所作主张的非正式推导，如果读者觉得这些推导冗长乏味，可以跳过。）
are written in italics. These sections constitute informal derivations of the claims made in the surrounding text and can be omitted by the reader who finds such derivations tedious.)
作为输入/输出模式 p 上的误差度量，并令 \E = \Σ_p E_p⟑ 为总体误差度量。我们希望证明，当单元是线性时，delta 规则实现了对 \E⟑ 的梯度下降。我们将简单地证明 \Δ_p w_{ji} ∝ -\partial E_p / \partial w_{ji}⟑，这与 delta 规则规定的 \Δ_p w_{ji}⟑ 成正比。当没有隐藏单元时，计算相关导数是直接的。为此
be our measure of the error on input/output pattern p and let \E = \Σ_p E_p⟑ be our overall measure of the error. We wish to show that the delta rule implements a gradient descent in \E⟑ when the units are linear. We will proceed by simply showing that \→Δ_p w_{ji} ∝ -\partial E_p / \partial w_{ji}⟑, which is proportional to \Δ_p w_{ji}⟑ as prescribed by the delta rule. When there are no hidden units it is straightforward to compute the relevant derivative. For this purpose
使用链式法则将导数写成两部分的乘积：误差对单元输出的导数乘以输出对权重的导数。
use the chain rule to write the derivative as the product of two parts: the derivative of the error with respect to the output of the unit times the derivative of the output with respect to the weight.
第一部分描述了误差如何随第 j 个单元的输出变化，第二部分则描述了改变 W_j 对输出的影响程度。现在，这些导数很容易计算。首先，从方程 2 出发
The first part tells how the error changes with the output of the j-th unit, and the second part tells how much changing W_j changes that output. Now, the derivatives are easy to compute. First, from Equation 2
毫不意外，单元 U_j 对误差的贡献简单地正比于 a p_j。
Not surprisingly, the contribution of unit U_j to the error is simply proportional to a p_j.
因此，代回方程 3，我们得到
Thus, substituting back into Equation 3, we see that
版权材料 (6) 3 24 基本机制
Copyrighted Material (6) 3 24 BASIC MECHANISMS
E. 实际上，这仅在权值在该周期内不变时才严格成立。通过每呈现一个样本后都更新权值，我们在一定程度上偏离了 E 的真正梯度下降。
E. In fact, this is strictly true only if the values of the weights are not changed during this cycle. By changing the weights after each pattern is presented we depart to some extent from a true gradient descent in E.
前馈网络中半线性激活函数的 Delta 规则。我们已经展示了标准的 Delta 规则本质上是针对线性激活函数的误差平方和实现梯度下降。此时，在没有隐藏单元的情况下，误差曲面呈碗状，只有一个极小值，因此梯度下降能保证
The delta rule for semilinear activation functions in feedforward networks. We have shown how the standard delta rule essentially implements gradient descent in sum-squared error for linear activation functions. In this case, without hidden units, the error surface is shaped like a bowl with only one minimum, so gradient descent is guaranteed
找到最佳权值集合。然而，存在隐藏单元时，如何计算导数变得不那么明显，且误差曲面并非向上凸，因此存在陷入局部极小值的危险。本章的主要理论贡献在于证明了存在一种高效的计算导数的方法；主要的实证贡献在于证明了在多种学习任务中，看似致命的局部极小值问题实际上无关紧要。在本章末尾，我们将展示广义 Delta 规则如何应用于任意网络，但首先，我们局限于分层前馈网络。在这些网络中，输入单元是底层，输出单元是顶层；中间可以有多层隐藏单元，但每个单元只能将其输出发送给更高层，且只能从更低层接收输入。
to find the best set of weights. With hidden units, however, it is not so obvious how to compute the derivatives, and the error surface is not concave upwards, so there is the danger of getting stuck in local minima. The main theoretical contribution of this chapter is to show that there is an efficient way of computing the derivatives. The main empirical contribution is to show that the apparently fatal problem of local minima is irrelevant in a wide variety of learning tasks. At the end of the chapter we show how the generalized delta rule can be applied to arbitrary networks, but, to begin with, we confine ourselves to layered feedforward networks. In these networks, the input units are the bottom layer and the output units are the top layer. There can be many layers of hidden units in between, but every unit must send its output to higher layers than its own and must receive its input
来自比自己更低的层。给定输入向量后，通过前向传播计算输出向量，该过程依次计算每一层的激活值，使用前面层已经计算出的激活值。由于我们主要关心将此结果推广到有隐藏单元的情形，并且由于第 2 章中概述的原因，隐藏
from lower layers than its own. Given an input vector, the output vector is computed by a forward pass which computes the activity levels of each layer in turn using the already computed activity levels in the earlier layers. Since we are primarily interested in extending this result to the case with hidden units and since, for reasons outlined in Chapter 2, hidden
具有线性激活函数的单元并不能带来优势，因此我们首先从
units with linear activation functions provide no advantage, we begin by
将我们的分析推广到一组称为半线性的非线性激活函数（见第二章）。半线性激活函数是指单元的激活输出是单元净输入的非递减且可微的函数。
Generalizing our analysis to the set of nonlinear activation functions which we call semilinear (see Chapter 2). A semilinear activation function is one in which the output of a unit is a nondecreasing and differentiable function of the net input to the unit.
其中，若单元 i 是输入单元，则\(o_i = i\)。因此，半线性激活函数是指...
Where \(o_i = i\) if unit i is an input unit. Thus, a semilinear activation function is one in which...
且\(f\)是可微且非递减的。广义 Delta 规则适用于由具有半线性激活函数的单元组成的网络。注意，线性阈值单元不满足该要求，因为它们的导数在阈值处无穷大，而在其他地方为零。
And \(f\) is differentiable and nondecreasing. The generalized delta rule works if the network consists of units having semilinear activation functions. Notice that linear threshold units do not satisfy the requirement because their derivative is infinite at the threshold and zero elsewhere.
为了得到 Delta 规则的正确推广，我们必须设定...
To get the correct generalization of the delta rule, we must set...
其中\(E\)是之前定义的相同平方和误差函数。与标准 Delta 规则一样，将这一导数视为由两部分乘积得到是有用的：一部分反映误差随单元净输入的变化，另一部分表示改变特定权重对净输入的影响。因此我们可以写出...
Where \(E\) is the same sum-squared error function defined earlier. As in the standard delta rule, it is again useful to see this derivative as resulting from the product of two parts: one part reflecting the change in error as a function of the change in the net input to the unit and one part representing the effect of changing a particular weight on the net input. Thus we can write...
通过方程 7，我们看到第二个因子是现在让我们定义
By Equation 7 we see that the second factor is Now let us define
(JO)（通过将其与方程 4 比较，注意到这与……的定义一致）
(JO) (By comparing this to Equation 4, note that this is consistent with the definition of
opj 用于原始线性单元的 delta 规则中，因为当单元 Uj 是线性时，Opj = netpj
opj used in the original delta rule for linear units since Opj = netpj when unit Uj is
线性。）因此方程 9 具有等价形式
linear.) Equation 9 thus has the equivalent form
这说明，为了在 E 中实现梯度下降，我们应该根据……来改变我们的权重
This says that to implement gradient descent in E we should make our weight changes according to
基本机制（版权材料 3.26）
Basic Mechanisms (Copyrighted Material 3.26)
就像标准的 Delta 规则一样。关键在于弄清楚网络中每个单元\U_j\的\delta_{pj}\应该是什么。有趣的结果（我们现在推导）是，存在一个简单的递归计算这些\delta\的方法，可以通过在网络中反向传播误差信号来实现。
Just as in the standard delta rule. The trick is to figure out what \delta_{pj}\ should be for each unit \U_j\ in the network. The interesting result, which we now derive, is that there is a simple recursive computation of these \delta\'s which can be implemented by propagating error signals backward through the network.
为了计算\delta_{pj} = -\frac{\partial E}{\partial net_{pj}}\，我们应用链式法则将此偏导数
To compute \delta_{pj} = -\frac{\partial E}{\partial net_{pj}}\, we apply the chain rule to write this partial derivative
写成两个因子的乘积，一个因子反映误差作为单元输出的函数的变化，另一个反映输出作为输入变化的函数的变化。因此，我们有
as the product of two factors, one factor reflecting the change in error as a function of the output of the unit and one reflecting the change in the output as a function of changes in the input. Thus, we have
让我们计算第二个因子。由方程 8 可知
Let us compute the second factor. By Equation 8 we see that
这仅仅是第 j 个单元的挤压函数在净输入 net_{pj}处的导数。为了计算第一个因子，我们考虑两种情况。首先，假设单元 U_j 是网络的输出单元。在这种情况下，根据 E_p 的定义，
which is simply the derivative of the squashing function for the jth unit, evaluated at the net input net_{pj} to that unit. To compute the first factor, we consider two cases. First, assume that unit U_j is an output unit of the network. In this case, it follows from the definition of E_p that
这与我们使用标准 delta 规则得到的结果相同。将两个因子代入公式 11，我们得到
which is the same result as we obtained with the standard delta rule. Substituting for the two factors in Equation 11, we get
对于任何输出单元 U_j。如果 U_j 不是输出单元，我们使用链式法则写为
for any output unit U_j. If U_j is not an output unit we use the chain rule to write
在这种情况下，将方程 12 中的两个因子代入得到
In this case, substituting for the two factors in Equation 12 yields
当 u 不是输出单元时。方程(3)和(14)给出了计算网络中所有单元δ的递归过程，这些δ随后用于根据方程(11)计算网络中的权重变化。该过程构成了针对半线性单元前馈网络的广义 delta 规则。
whenever u is not an output unit. Equations (3) and (14) give a recursive procedure for computing the δ's for all units in the network, which are then used to compute the weight changes in the network according to Equation (11). This procedure constitutes the generalized delta rule for a feedforward network of semilinear units.
这些结果可以归纳为三个方程。首先，广义 delta 规则与方程 1 的标准 delta 规则形式完全相同。每条线上的权重应该改变一个量，该量与沿着该线接收输入的单元可获得的误差信号δ以及沿着该线发送激活的单元的输出成正比。用符号表示，
These results can be summarized in three equations. First, the generalized delta rule has exactly the same form as the standard delta rule of Equation 1. The weight on each line should be changed by an amount proportional to the product of the error signal, δ, available to the unit receiving input along that line and the output of the unit sending activation along that line. In symbols,
沿着该线接收输入的单元和沿着该线发送激活的单元的输出。用符号表示，另外两个方程明确了误差信号。本质上，误差信号的确定是一个递归过程，从输出单元开始。如果一个单元是输出单元，其误差信号与标准 delta 规则非常相似。由下式给出：
the unit receiving input along that line and the output of the unit sending activation along that line. In symbols, the other two equations specify the error signal. Essentially, the determination of the error signal is a recursive process which starts with the output units. If a unit is an output unit, its error signal is very similar to the standard delta rule. It is given by
其中\(f'_j(net_{pj})\)是半线性激活函数的导数，该函数将单元的总输入映射到输出值。最后，对于没有指定目标的隐藏单元，其误差信号通过递归方式根据其直接连接的单元的误差信号和这些连接的权重来确定。即，
where \(f'_j(net_{pj})\) is the derivative of the semilinear activation function which maps the total input to the unit to an output value. Finally, the error signal for hidden units for which there is no specified target is determined recursively in terms of the error signals of the units to which it directly connects and the weights of those connections. That is,
每当单元不是输出单元时。因此，广义 delta 规则的应用包括两个阶段：在第一阶段，输入被呈现并通过网络向前传播，以计算每个单元的输出值\(O_{pj}\)。然后将此输出与目标进行比较，得到每个输出单元的误差信号\(δ_{pj}\)。第二阶段涉及通过网络的一次反向传播（类似于初始的前向传播），在此期间，误差信号被传递到网络中的每个单元，并进行适当的权重更改。这第二次反向传播允许如上所述的δ的递归计算。第一步是计算每个输出单元的δ。这仅仅是实际输出值与期望输出值之差乘以挤压函数的导数。然后我们可以计算所有馈入最终层的连接的权重变化。完成之后，再计算倒数第二层所有单元的δ。这样将误差向后传播一层，并且可以对每一层重复相同的过程。反向传播的计算复杂度与前向传播相同，因此不会过于昂贵。我们现在已经生成了一种梯度下降方法，用于寻找任何具有半线性单元的前馈网络中的权重。在报告我们使用这些网络的结果之前，注意一些进一步的观察是很有用的。有趣的是，并非所有权重都必须是可变的。网络中的任意数量的权重都可以固定。在这种情况下，误差仍然通过固定权重传播；固定权重只是不更新。
whenever the unit is not an output unit. The application of the generalized delta rule, thus, involves two phases: During the first phase the input is presented and propagated forward through the network to compute the output value \(O_{pj}\) for each unit. This output is then compared with the targets, resulting in an error signal \(δ_{pj}\) for each output unit. The second phase involves a backward pass through the network (analogous to the initial forward pass) during which the error signal is passed to each unit in the network and the appropriate weight changes are made. This second, backward pass allows the recursive computation of δ as indicated above. The first step is to compute δ for each of the output units. This is simply the difference between the actual and desired output values times the derivative of the squashing function. We can then compute weight changes for all connections that feed into the final layer. After this is done, then compute δ's for all units in the penultimate layer. This propagates the errors back one layer, and the same process can be repeated for every layer. The backward pass has the same computational complexity as the forward pass, and so it is not unduly expensive. We have now generated a gradient descent method for finding weights in any feedforward network with semilinear units. Before reporting our results with these networks, it is useful to note some further observations. It is interesting that not all weights need be variable. Any number of weights in the network can be fixed. In this case, error is still propagated through the fixed weights; the fixed weights are simply not updated.
需要修改。还应指出，没有理由认为某些输出单元不能从早期层的其他输出单元接收输入。在这种情况下，这些单元接收两种不同的误差：一种来自与目标的直接比较，另一种来自通过其激活影响的其他输出单元传递的误差。在这种情况下，正确的做法是简单地将直接比较所决定的权重变化与从其他输出单元反向传播的权重变化相加。
Modified. It should also be noted that there is no reason why some output units might not receive inputs from other output units in earlier layers. In this case, those units receive two different kinds of error: that from the direct comparison with the target and that passed through the other output units whose activation it affects. In this case, the correct procedure is to simply add the weight changes dictated by the direct comparison to that propagated back from the other output units.
我们现在有了一个学习过程，原则上可以演化出一组权重，以产生从输入到输出的任意映射。然而，我们提出的过程是一个梯度下降过程，因此受到任何爬山过程的所有问题的制约——即局部极大值问题，或者（在我们的情况下）
We now have a learning procedure which could, in principle, evolve a set of weights to produce an arbitrary mapping from input to output. However, the procedure we have produced is a gradient descent procedure and, as such, is bound by all of the problems of any hill climbing procedure — namely, the problem of local maxima or (in our case)
极小值。此外，还有一个问题，即系统学习需要多长时间。即使我们能保证它最终会找到解，也存在我们的过程能否在合理的时间内学习的问题。有趣的是，系统在解决特定问题时实际发展出了哪些隐藏单元。这实际上是系统创建了何种内部表示的问题。我们尚未对这些问题给出明确答案。然而，我们进行了许多模拟，这些模拟使我们对局部极小值和时间问题感到乐观，并对我们的学习机制所发现的表示类型感到惊讶。在继续呈现我们的结果之前，我们必须更详细地描述我们的模拟系统。特别是，我们必须指定一个激活函数，并展示系统如何计算该函数的导数。
minima. Moreover, there is a question of how long it might take a system to learn. Even if we could guarantee that it would eventually find a solution, there is the question of whether our procedure could learn in a reasonable period of time. It is interesting to ask what hidden units the system actually develops in the solution of particular problems. This is the question of what kinds of internal representations the system actually creates. We do not yet have definitive answers to these questions. However, we have carried out many simulations which lead us to be optimistic about the local minima and time questions and to be surprised by the kinds of representations our learning mechanism discovers. Before proceeding with our results, we must describe our simulation system in more detail. In particular, we must specify an activation function and show how the system can compute the derivative of this function.
一个有用的激活函数。在我们上述推导中，单元 u 的激活函数的导数\(r_j(\text{net}\)\)始终发挥作用。这意味着我们需要一个存在导数的激活函数。值得注意的是，感知机所基于的线性阈值函数是不连续的，因此不适用于广义 delta 规则。类似地，由于线性系统无法从隐藏单元获得优势，线性激活函数也不适用。因此，我们需要一个连续的非线性激活函数。在大多数实验中，我们使用了 logistic
A useful activation function. In our above derivations the derivative of the activation function of unit u, \(r_j(\text{net}\)\), always played a role. This implies that we need an activation function for which a derivative exists. It is interesting to note that the linear threshold function, on which the perceptron is based, is discontinuous and hence will not suffice for the generalized delta rule. Similarly, since a linear system achieves no advantage from hidden units, a linear activation function will not suffice either. Thus, we need a continuous, nonlinear activation function. In most of our experiments we have used the logistic
激活函数在 w 中。8. 学习内部表示 329
activation function in w. 8. LEARNING INTERNAL REPRESENTATIONS 329
其中 \θ_j\ 是一个在功能上类似于阈值的偏置。为了应用我们的学习规则，我们需要知道该函数关于其总输入 \net_{pj}\ 的导数，其中 \net_{pj} = \sum_i w_{ji} o_{pi} + \theta_j\。很容易证明该导数由下式给出
where \θ_j\ is a bias similar in function to a threshold. In order to apply our learning rule, we need to know the derivative of this function with respect to its total input, \net_{pj}\, where \net_{pj} = \sum_i w_{ji} o_{pi} + \theta_j\. It is easy to show that this derivative is given by
因此，对于 logistic 激活函数，输出单元的误差信号 \\delta_{pj}\ 由下式给出
Thus, for the logistic activation function, the error signal, \\delta_{pj}\, for an output unit is given by
并且任意隐藏单元 \U_j\ 的误差由下式给出
and the error for an arbitrary hidden unit \U_j\ is given by
需要注意的是，导数 \o_{pj} (1 - o_{pj})\ 在 \o_{pj} = 0.5\ 时达到最大值，并且由于 \0 \le o_{pj} \le 1\，当 \o_{pj}\ 接近 0 或 1 时趋近于最小值
It should be noted that the derivative, \o_{pj} (1 - o_{pj})\, reaches its maximum
where () j is a b ias similar i n fu nction to a t hreshold . 1 I n order to apply our l earning r u l e, we need t o k now the derivative of t h i s fu nction wit h respect to i ts total i nput, netpj, where netpj = L, wJ; op; + () J. It is easy to show that t h is derivative is given by
接近于 0 或 1。由于给定权重的变化量与该导数成正比，权重的变化对于接近其中间范围且尚未完全确定的单元来说是最大的。
approaches zero or one. Since the amount of change in a given weight is proportional to this derivative, weights will be changed most for those units that are near their mid range and, in some sense, not yet
完全处于开启或关闭状态。我们相信，这一特性有助于系统学习的稳定性。还需注意该激活函数的另一个特性：系统实际上无法达到 1 或 0 的极值，除非权
committed to being either on or off. This feature, we believe, contributes to the stability of the learning of the system. One other feature of this activation function should be noted. The system cannot actually reach its extreme values of 1 or 0 without infin
重无穷大。因此，在期望输出为二值（0,1）的实际学习场景中，系统永远无法真正达到这些值。因此，我们通常使用 0.1 和 0.9 作为目标值，尽管我们在讨论时仍会假设追求的是（0,1）值。
itely large weights. Therefore, in a practical learning situation in which the desired outputs are binary (0,1), the system can never actually achieve these values. Therefore, we typically use the values of 0.1 and 0.9 as the targets, even though we will talk as if values of (0,1) are sought.
学习率。我们的学习过程仅要求权重的变化与∂E/∂w 成正比。真正的梯度下降需要采取无穷小的步长。比例常数就是我们过程中的学习率。该常数越大，权重的变化越大。实际应用中，我们选择一个
The learning rate. Our learning procedure requires only that the change in weight be proportional to ∂E/∂w. True gradient descent requires that infinitesimal steps be taken. The constant of proportionality is the learning rate in our procedure. The larger this constant, the larger the changes in the weights. For practical purposes we choose a
版权所有。330 基本机制
Copyrighted Material. 330 BASIC MECHANISMS
尽可能大的学习率而不导致振荡。这提供了最快的学习。一种在不导致振荡的情况下增加学习率的方法是修改广义 delta 规则以包含动量项。这可以通过以下规则实现：(16) 其中下标 n 表示呈现次数序号，η是学习率，α是决定过去权重变化对当前权重空间移动方向影响的常数。这就在权重空间中提供了一种动量，有效地过滤了误差曲面在权重空间中的高频变化。这在包含长峡谷的空间中很有用，这些峡谷的特征是横跨峡谷的陡峭曲率和缓坡底部。陡峭曲率往往会引起横跨峡谷的发散振荡。为了防止这些振荡，必须采用非常小的步长，但这会导致沿峡谷前进非常缓慢。动量过滤了高曲率，从而允许更大的有效权重步长。在我们的大多数模拟中，α约为 0.9。我们的经验是，通过设置α=0 并减小步长大小，我们可以得到相同的解。
learning rate that is as large as possible without leading to oscillation. This offers the most rapid learning. One way to increase the learning rate without leading to oscillation is to modify the generalized delta rule to include a momentum term. This can be accomplished by the following rule: (16) where the subscript n indexes the presentation number, η is the learning rate, and α is a constant which determines the effect of past weight changes on the current direction of movement in weight space. This provides a kind of momentum in weight space that effectively filters out high-frequency variations of the error-surface in the weight space. This is useful in spaces containing long ravines that are characterized by sharp curvature across the ravine and a gently sloping floor. The sharp curvature tends to cause divergent oscillations across the ravine. To prevent these it is necessary to take very small steps, but this causes very slow progress along the ravine. The momentum filters out the high curvature and thus allows the effective weight steps to be bigger. In most of our simulations α was about 0.9. Our experience has been that we get the same solutions by setting α = 0 and reducing the size of
但使用较大的α值，系统整体学习速度更快。
but the system learns much faster overall with larger values of α
对称性破缺。我们的学习过程还有一个容易克服的问题，即对称性破缺问题。如果所有权重都以相等的值开始，并且解决方案需要发展出不相等的权重，那么系统永远无法学习。这是因为误差通过权重反向传播，与权重值成比例。这意味着所有直接连接到输出输入的隐藏单元将获得相同的误差信号，而且由于权重变化依赖于误差信号，从这些单元到输出单元的权重必须始终相同。系统开始于一种局部最大值，它保持权重相等，但它是误差函数的最大值，因此一旦逃离就永远不会返回。我们通过以小的随机权重启动系统来抵消这个问题。在这种情况下，此类对称性问题不会出现。
Symmetry breaking. Our learning procedure has one more problem that can be readily overcome and this is the problem of symmetry breaking. If all weights start out with equal values and if the solution requires that unequal weights be developed, the system can never learn. This is because error is propagated back through the weights in proportion to the values of the weights. This means that all hidden units connected directly to the output inputs will get identical error signals, and, since the weight changes depend on the error signals, the weights from those units to the output units must always be the same. The system is starting out at a kind of local maximum, which keeps the weights equal, but it is a maximum of the error function, so once it escapes it will never return. We counteract this problem by starting the system with small random weights. Under these conditions symmetry problems of this kind do not arise.
从异或问题开始是有用的，因为它是需要隐藏单元的经典问题，而且许多其他困难
It is useful to begin with the exclusive-or problem since it is the classic problem requiring hidden units and since many other difficult
learning rate that is as large as possi ble wi thout leading to osci l lation . This offers the most rapid learni ng. One way to increase the learning rate without leading to oscillation is to modify the general ized delta rule to include a momentum term. This can be accomplished by the follow ing rule: (16) where the subscri pt n i ndexes the presentation n umber, 'T/ i s the learn i ng rate, and a is a constant which determi nes the effect of past weight changes on the current di rection of movement in weight space . This provides a kind of momentum i n weight space that effecti vely fi lters out high-frequency variations of the error-su rface i n the weight space. This is useful i n spaces contai ning long ravi nes that are characterized by sharp curvature across the ravi ne and a gently sloping floor. The sharp curvature tends to cause divergen t osci l l ations across the ravine. To prevent these i t is necessary to take very small steps, but this causes very slow progress along the ravi ne. The momentum fi l ters out the high curvat ure and thus allows the effective weight steps to be bi gger. In most of our sim ulations a was about 0.9. Our experience has been that we get the same soluti ons by setti ng a = 0 and reducing the size of
问题涉及一个异或（XOR）子问题。我们已经运行了该 XOR
Problems involve an XOR as a subproblem. We have run the XOR
问题多次，除了下面讨论的几个例外，系统总能解决该问题。图 3 展示了该问题的一个解。这个解是在以 0.5 的学习率遍历四个刺激模式 558 次后得到的。在这种情况下，隐藏单元和输出单元都有正偏置
problem many times and with a couple of exceptions discussed below, the system has always solved the problem. Figure 3 shows one of the solutions to the problem. This solution was reached after 558 sweeps through the four stimulus patterns with a learning rate of 0.5. In this case, both the hidden unit and the output unit have positive biases
所以它们默认开启，除非被关闭。如果两个输入单元都没有开启，隐藏单元就会开启。当它开启时，它会关闭输出单元。连
so they are on unless turned off. The hidden unit turns on if neither input unit is on. When it is on, it turns off the output unit. The con
接到输出单元的连接自行安排，使得每当两个输入都开启时，它们就关闭输出单元。在这种情况下，网络稳定到了一个解，该解是图 2 所示解的一种镜像。
nections from input to output units arranged themselves so that they turn off the output unit whenever both inputs are on. In this case, the network has settled to a solution which is a sort of mirror image of the one illustrated in Figure 2.
我们已经训练系统解决 XOR 问题数百
We have taught the system to solve the XOR problem hundreds of
次。有时我们使用一个隐藏单元和直接连接
times. Sometimes we have used a single hidden unit and direct connections
连接到输出单元，如下图所示；其他时候我们允许两个隐藏单元，并将从输入单元到输出的连接设为零，如图 4 所示。只有两次系统遇到了局部最小值，因此无法解决问题。这两种情况都涉及两个隐藏单元的版本。
connections to the output unit as illustrated here, and other times we have allowed two hidden units and set the connections from the input units to the outputs to be zero, as shown in Figure 4. In only two cases has the system encountered a local minimum and thus been unable to solve the problem. Both cases involved the two hidden units version of the
一个简单的架构，使用两个隐藏单元且没有从输入到输出的直接连接来解决 XOR 问题。
A simple architecture for solving XOR with two hidden units and no direct connections from input to output.
问题，两者最终都陷入了同一个局部最小值。图 5
problem and both ended up in the same local minimum. Figure 5
显示了局部最小值的权重。在这种情况下，系统正确响应了两个模式——即模式 00 和 10。而对于另外两个模式 11 和 01，输出单元的净输入为零。这导致这两个模式的输出值都为 0.5。这个状态是在每个模式呈现 6587 次、学习率\(\eta=0.25\)后达到的。尽管许多问题需要更多次的呈现才能学习，但对该问题的进一步尝试仅仅增加了权重的大小，并没有提高性能。我们不知道这种局部最小值的频率，但我们在该问题和其他问题上的经验是它们非常罕见。我们只发现了另一种情况，其中
shows the weights for the local minimum. In this case, the system correctly responds to two of the patterns—namely, the patterns 00 and 10. In the cases of the other two patterns 11 and 01, the output unit gets a net input of zero. This leads to an output value of 0.5 for both of these patterns. This state was reached after 6,587 presentations of each pattern with \(\eta=0.25\). 2 Although many problems require more presentations for learning to occur, further trials on this problem merely increase the magnitude of the weights but do not lead to any improvement in performance. We do not know the frequency of such local minima, but our experience with this and other problems is that they are quite rare. We have found only one other situation in which a
局部最小值出现在数百个各种类型的问题中。我们将在下面讨论这种情况。XOR 问题已被证明是许多其他研究的有用测试案例。使用图 4 所示的架构，一位学生
local minimum has occurred in many hundreds of problems of various sorts. We will discuss this case below. The XOR problem has proved a useful test case for a number of other studies. Using the architecture illustrated in Figure 4, a student
ple architecture for solving XOR with two hi dde n units and no direct connections from input to output.
我们通常将η设为 0.5 或以上以避免局部最小值。然而，一般情况下，
we set η = 0.5 or above to avoid local minima. In general, however,
避免局部最小值的方法是使用非常小的η值。§8. 学习内部表示 333
way to avoid local minima is to use very small values of η. §8. Learning Internal Representations 333
隐藏单元数量以及随时间调整学习率以解决问题。使用每个模式 0.01 的误差作为学习标准，Yves 发现使用η=0.25 解决问题所需的平均呈现次数从两个隐藏单元时的约 245 次变化到 32 个隐藏单元时的约 120 次。结果可总结为\(P = 280 - 33 \log_2 H\)，其中 P 是所需的呈现次数，H 是使用的隐藏单元数量。因此，解决 XOR 问题的时间随隐藏单元数量的对数线性减少。这一结果适用于
number of hidden units and varying the learning rate over time to solve the problem. Using as a learning criterion an error of 0.01 per pattern, Yves found that the average number of presentations to solve the problem with η = 0.25 varied from about 245 for the case with two hidden units to about 120 presentations for 32 hidden units. The results can be summarized by \(P = 280 - 33 \log_2 H\), where P is the required number of presentations and H is the number of hidden units employed. Thus, the time to solve XOR is reduced linearly with the logarithm of the number of hidden units. This result holds for values of
H 最多到 40（在 XOR 情况下）。通过增加隐藏单元数量可以缩短求解时间，这一普遍结果在我们几乎所有的模拟中都观察到了。Yves 还研究了八个隐藏单元情况下求解时间与学习率的关系。他发现η=0.1 时平均约 450 次呈现。
H up to about 40 in the case of XOR. The general result that the time to solution is reduced by increasing the number of hidden units has been observed in virtually all of our simulations. Yves also studied the time to solution as a function of learning rate for the case of eight hidden units. He found an average of about 450 presentations with η = 0.1.
η=0.75 时，他发现平均约 68 次呈现。他还发现
With η = 0.75, he found an average of about 68 presentations. He also found that
版权资料 3 34 基本机制
Copyrighted Material 3 34 BASIC MECHANISMS
超过此值的学习率会导致不稳定行为。然而，在此范围内，较大的学习率显著加快了学习速度。在大多数问题中，我们采用了η = 0.25 的学习率。
Learning rates larger than this led to unstable behavior. However, within this range larger learning rates speeded the learning substantially. In most of our problems we have employed learning rates of η = 0.25
Minsky 和 Papert（1969）深入讨论的问题之一是奇偶问题，其中如果输入模式包含奇数个 1，则所需输出为 1，否则为 0。这是一个非常困难的问题，因为最相似的模式（只有一位不同）需要不同的答案。异或问题是一个输入模式大小为二的奇偶问题。我们尝试了
One of the problems given a good deal of discussion by Minsky and Papert (1969) is the parity problem, in which the output required is 1 if the input pattern contains an odd number of 1s and 0 otherwise. This is a very difficult problem because the most similar patterns (those which differ by a single bit) require different answers. The XOR problem is a parity problem with input patterns of size two. We have tried a
一系列奇偶问题，模式大小从二到八不等。通常我们采用分层网络，其中不允许从输入到输出单元的直接连接，必须通过一组隐藏单元进行中介。在这种架构中，解决长度为 N 的模式奇偶问题至少需要 N 个隐藏单元。图 6 展示了系统发现的解的基本范式。图中实线表示权重+1，虚线表示权重-1。圆圈中的数字表示单元的偏置。基本上，隐藏单元排列
number of parity problems with patterns ranging from size two to eight. Generally we have employed layered networks in which direct connections from the input to the output units are not allowed, but must be mediated through a set of hidden units. In this architecture, it requires at least N hidden units to solve parity with patterns of length N. Figure 6 illustrates the basic paradigm for the solutions discovered by the system. The solid lines in the figure indicate weights of +1 and the dotted lines indicate weights of -1. The numbers in the circles represent the biases of the units. Basically, the hidden units arranged
版权资料 8. 学习内部表示 3 3 5 它们自身如此排列以便计数输入的数量。在图中，最左边的一个在有一个或多个输入单元开启时开启，下一个在两个或更多开启时开启，依此类推。如果所有输入线都开启，则所有隐藏单元都开启。当输入模式中有 m 个位开启时，前 m 个隐藏单元开启。然后隐藏单元以交替的正负权重连接。这样，对于偶数个输入，隐藏单元的净输入为零，对于奇数个输入则为+1。表 3 展示了我们对四个输入线和四个隐藏单元进行模拟所获得的实际解。这个解是在η=0.5 的情况下，对十六个模式每个呈现 2,825 次后达到的。注意，该解大致是图 6 所示解的镜像，因为开启的隐藏单元数量等于零输入值的数量，而不是 1 的数量。除此之外，原理如上所示。应该指出，学习规则创建的内部表示是安排开启的隐藏单元数量等于输入中零的数量，并且开启的特定隐藏单元只取决于数量，而不取决于哪些输入单元开启。这正是奇偶问题所需的那种重新编码。它不是那种容易被无监督学习方案（如竞争学习）发现的表示。
Copyrighted Material 8. LEARNING INTERNAL REPRESENTATIONS 3 3 5 themselves so that they count the number of inputs. In the diagram, the one at the far left comes on if one or more input units are on, the next comes on if two or more are on, etc. All of the hidden units come on if all of the input lines are on. The first m hidden units come on whenever m bits are on in the input pattern. The hidden units then connect with alternately positive and negative weights. In this way the net input from the hidden units is zero for even numbers and +1 for odd numbers. Table 3 shows the actual solution attained for one of our simulations with four input lines and four hidden units. This solution was reached after 2,825 presentations of each of the sixteen patterns with η = 0.5. Note that the solution is roughly a mirror image of that shown in Figure 6 in that the number of hidden units turned on is equal to the number of zero input values rather than the number of ones. Beyond that the principle is that shown above. It should be noted that the internal representation created by the learning rule is to arrange that the number of hidden units that come on is equal to the number of zeros in the input and that the particular hidden units that come on depend only on the number, not on which input units are on. This is exactly the sort of recoding required by parity. It is not the kind of representation readily discovered by unsupervised learning schemes such as competitive learning.
Ackley、Hinton 和 Sejnowski (1985) 提出了一个问题：通过少量隐藏单元将一组正交输入模式映射到一组正交输出模式。在这种情况下，隐藏单元上的模式内部表示必须相当高效。假设我们尝试将 N 个输入模式映射到
Ackley, Hinton, and Sejnowski (1985) have posed a problem in which a set of orthogonal input patterns are mapped to a set of orthogonal output patterns through a small set of hidden units. In such cases the internal representations of the patterns on the hidden units must be rather efficient. Suppose that we attempt to map N input patterns onto
输出模式。进一步假设提供了\(\log_2 N\)个隐藏单元。在这种情况下，我们期望系统学会使用
output patterns. Suppose further that \(\log_2 N\) hidden units are provided. In this case, we expect that the system will learn to use the
N 个输出单元 log N 个隐藏单元 2N 个输入单元 图 7. 解决编码器问题的网络。在这个问题中有 N 个
N Output Units log N Hidden Units 2N Input Units FIGURE 7. A network for solving the encoder problem. In this problem there are N
N 个正交输出模式。只有\(\log_2 N\)个隐藏单元。因此，如果隐藏单元取二进制值，那么隐藏单元必须形成一个二进制数来编码每个输入模式。这正是
N orthogonal output patterns. There are only \(\log_2 N\) hidden units. Thus, if the hidden units take on binary values, the hidden units must form a binary number to encode each of the input patterns. This is exactly
隐藏单元形成一个二进制码，对 N 个输入模式中的每一个都有一个不同的二进制模式。图 7 展示了编码器问题的基本架构。本质上，问题是学习将一个 N 位模式编码为一个\(\log_2 N\)位模式，然后学习将这个表示解码为输出模式。我们向系统展示了许多这类问题。这里我们展示一个有八个输入模式、八个输出模式和三个隐藏单元的问题。在这种情况下，要求的映射是表 4 所示的恒等映射。问题简单地是开启相同的位在
hidden units to form a binary code with a distinct binary pattern for each of the N input patterns. Figure 7 illustrates the basic architecture for the encoder problem. Essentially, the problem is to learn an encoding of an N bit pattern into a \(\log_2 N\) bit pattern and then learn to decode this representation into the output pattern. We have presented the system with a number of these problems. Here we present a problem with eight input patterns, eight output patterns, and three hidden units. In this case the required mapping is the identity mapping illustrated in Table 4. The problem is simply to turn on the same bit in the
10000000 01000000 00100000 00010000 00001000 00000100 00000010 00000001
10000000 01000000 00100000 00010000 00001000 00000100 00000010 00000001
10000000 01000000 00100000 00010000 00001000 00000100 00000010 00000001
10000000 01000000 00100000 00010000 00001000 00000100 00000010 00000001
版权所有资料 8. 学习内部表示 337
Copyrighted Material 8. LEARNING INTERNAL REPRESENTATIONS 337
输出与输入相同。表 5 展示了我们的学习系统在该示例上生成的映射。有趣的是，系统运用了其使用中间值的能力来解决这个问题。当然，它本可以找到一个隐藏单元只取 0 和 1 值的解。通常它就是这样做的，但在本例以及许多其他情况下，存在使用中间值的解，并且学习系统找到了它们，尽管它对极端值存在偏好。可以设置一些问题，使得
output as in the input. Table 5 shows the mapping generated by our learning system on this example. It is of some interest that the system employed its ability to use intermediate values in solving this problem. It could, of course, have found a solution in which the hidden units took on only the values of zero and one. Often it does just that, but in this instance, and many others, there are solutions that use the intermediate values, and the learning system finds them even though it has a bias toward extreme values. It is possible to set up problems that
要求系统利用中间值来解决问题。我们现在转向这样一个案例。表 6 展示了一个非常简单的问题，我们需要将两个单元上的分布式表示转换为局部表示。
require the system to make use of intermediate values in order to solve a problem. We now turn to such a case. Table 6 shows a very simple problem in which we have to convert from a distributed representation over two units into a local representation
在四个单元上。分布式输入模式的相似性结构在局部输出表示中根本未被保留。我们向学习系统提出了这个问题，并施加了若干约束，使其尤为困难。两个输入单元仅允许连接到单个隐藏单元，而该隐藏单元又允许连接到另外四个隐藏单元。只有这四个隐藏单元允许连接到四个输出单元。因此，要解决这个问题，系统必须首先将分布式
over four units. The similarity structure of the distributed input patterns is simply not preserved in the local output representation. We presented this problem to our learning system with a number of constraints which made it especially difficult. The two input units were only allowed to connect to a single hidden unit which, in turn, was allowed to connect to four more hidden units. Only these four hidden units were allowed to connect to the four output units. To solve this problem, then, the system must first convert the distributed
输入模式的表示转换为单个隐藏单元的各种中间值，其中不同的激活值对应不同的输入模式。然后，这些连续值必须通过下一层隐藏单元先转换回另一个分布式表示，最后转换为局部表示。该问题被呈现给系统，经过 5,226 次呈现、学习率η=0.05 后，系统找到了解决方案。表 7
representation of the input patterns into various intermediate values of the singleton hidden unit in which different activation values correspond to the different input patterns. These continuous values must then be converted back through the next layer of hidden units first to another distributed representation and then, finally, to a local representation. This problem was presented to the system and it reached a solution after 5,226 presentations with η = 0.05. Table 7
展示了系统实际为转换模式并解决问题而发展的表示序列。注意，四个输入模式中的每一个都被映射到单个隐藏单元的特定激活值上。这些值随后被映射到下一层隐藏单元的分布式模式，最终映射到输出层所需的局部表示。原则上，这种将模式映射到激活值、再将激活值映射回模式的技巧可用于任意数量的模式，但随着必须区分的激活值差异越来越小，系统做出必要区分也变得越来越困难。图 8 展示了系统为完成此任务而发展的网络。为清晰起见，从隐藏单元到输出单元的连接权重已被省略（但连接符号通过连接形式指示，例如虚线表示抑制连接）。四个不同的激活值是通过使用符号相反且相对较大的权重生成的。一条输入线将隐藏单元完全打开，另一条将其完全关闭。两者相差相对较小，因此当两者都打开时，该单元获得介于 0
shows the sequence of representations the system actually developed in order to transform the patterns and solve the problem. Note each of the four input patterns was mapped onto a particular activation value of the singleton hidden unit. These values were then mapped onto distributed patterns at the next layer of hidden units which were finally mapped into the required local representation at the output level. In principle, this trick of mapping patterns into activation values and then converting those activation values back into patterns could be done for any number of patterns, but it becomes increasingly difficult for the system to make the necessary distinctions as ever smaller differences among activation values must be distinguished. Figure 8 shows the network the system developed to do this job. The connection weights from the hidden units to the output units have been suppressed for clarity. (The sign of the connection, however, is indicated by the form of the connection - e.g., dashed lines mean inhibitory connections). The four different activation values were generated by having relatively large weights of opposite sign. One input line turns the hidden unit full on, one turns it full off. The two differ by a relatively small amount so that when both turn on, the unit attains a value intermediate between 0
和 0.5 之间的值。当两者都未打开时，接近零的偏置使单元获得略高于 0.5 的值。到第二层隐藏单元的连接同样有趣。当隐藏单元完全打开时，
and 0.5. When neither turns on, the near zero bias causes the unit to attain a value slightly over 0.5. The connections to the second layer of hidden units is likewise interesting. When the hidden unit is full on,
版权材料 8. 学习内部表示
Copyrighted Material 8. LEARNING INTERNAL REPRESENTATIONS
输出单元 隐藏单元 输入单元
Output Units Hidden Units Input Units
这些隐藏单元中最右边的一个被开启，所有其他单元被关闭。当隐藏单元关闭时，其他三个隐藏单元开启，最左边单元关闭。从单个隐藏单元到其他隐藏单元的其他连接是渐变的，以便为其另外两个值开启不同的模式。这里我们有一个学习系统灵活性的例子。我们的经验是，隐藏单元倾向于取极值，但是，只要学习问题需要，它们可以学会取渐变值。这种取极值的倾向很可能源于 Logistic 函数是 Sigmoid 函数，因此输入幅值增大会将其推向 0 或 1。这意味着，在需要中间值的问题中，输入权重必须保持适中大小。有趣的是，广义 Delta 规则的推导并不依赖于所有单元具有相同的激活函数。因此，某些需要以渐变方式编码信息的单元可以是线性的，而其他单元可以是 Logistic 的。线性单元将具有更宽的动态范围，并可以编码更多不同的值。这将是线性单元在具有硬性材料 340 基本机制的网络中的有用角色。
The right-most of these hidden units is turned on and all others turned off. When the hidden unit is turned off, the other three of these hidden units are on and the left-most unit off. The other connections from the singleton hidden unit to the other hidden units are graded so that a distinct pattern is turned on for its other two values. Here we have an example of the flexibility of the learning system. Our experience is that there is a propensity for the hidden units to take on extreme values, but, whenever the learning problem calls for it, they can learn to take on graded values. It is likely that the propensity to take on extreme values follows from the fact that the logistic is a sigmoid so that increasing magnitudes of its inputs push it toward zero or one. This means that in a problem in which intermediate values are required, the incoming weights must remain of moderate size. It is interesting that the derivation of the generalized delta rule does not depend on all of the units having identical activation functions. Thus, it would be possible for some units, those required to encode information in a graded fashion, to be linear while others might be logistic. The linear unit would have a much wider dynamic range and could encode more different values. This would be a useful role for a linear unit in a network with hard material 340 BASIC MECHANISMS
我们研究的另一个有趣问题是，根据输入字符串是否关于中心对称对其进行分类。我们使用了不同长度的模式和不同数量的隐藏单元。令我们惊讶的是，我们发现该问题总是仅用两个隐藏单元就能解决。为了理解导出的表示，考虑我们的系统为长度为 6 的字符串生成的一个解。该解是在每个六位模式呈现 1,208 次、η = 0.1 后得到的。最终网络如图 9 所示。
Another interesting problem we studied is that of classifying input strings as to whether or not they are symmetric about their center. We used patterns of various lengths with various numbers of hidden units. To our surprise, we discovered that the problem can always be solved with only two hidden units. To understand the derived representation, consider one of the solutions generated by our system for strings of length six. This solution was arrived at after 1,208 presentations of each six-bit pattern with η = 0.1. The final network is shown in Figure 9.
为简单起见，我们将六个输入单元显示在图的中央，一个隐藏单元在上方，一个在下方。输出单元在最右侧，指示字符串是否关于其中心对称。关于这个解，需要注意的关键点是，对于给定的隐藏单元，关于中间对称的权重大小相等、符号相反。这意味着，如果对称模式开启，两个隐藏单元将从输入单元接收到净输入为零，并且由于隐藏单元具有负偏置，两者都将关闭。在这种情况下，具有正偏置的输出单元，
For simplicity we have shown the six input units in the center of the diagram with one hidden unit above and one below. The output unit, which signals whether or not the string is symmetric about its center, is shown at the far right. The key point to see about this solution is that for a given hidden unit, weights that are symmetric about the middle are equal in magnitude and opposite in sign. That means that if a symmetric pattern is on, both hidden units will receive a net input of zero from the input units, and, since the hidden units have a negative bias, both will be off. In this case, the output unit, having a positive bias,
9. 解决对称性问题的网络。六个空心圆代表输入单元。有两个隐藏单元，一个显示在输入单元上方，一个显示在下方。输出单元显示在最右边。详见正文。
9. Network for solving the symmetry problem. The six open circles represent the input units. There are two hidden units, one shown above and one below the input units. The output unit is shown to the far right. See text for explanation.
版权材料 8. 学习内部表示 341
Copyrighted Material 8. LEARNING INTERNAL REPRESENTATIONS 341
这一单元将接通。关于该解，需要注意的第二重要的事情是，字符串中点两侧的权重之比为 1:2:4。这确保中点每一侧可能出现的八种模式中的每一种都向隐藏单元发送唯一的激活和。这保证了左侧没有任何模式能够恰好平衡右侧的非镜像模式。最后，两个隐藏单元从输入单元获得的权重模式除符号外完全相同。这确保了对于每一个非对称模式，至少一个隐藏单元会接通并激活输出单元。总之，网络的安排使得当模式对称时，两个隐藏单元从输入单元接收到的激活恰好为零，而对于每个非对称模式，至少一个隐藏单元会接收到正输入。这个问题之所以令我们感兴趣，是因为学习系统发展出了一个比我们先前考虑过的优雅得多的解。这个问题并非唯一出现这种情况的案例。学习过程发现的奇偶校验解也是我们在用学习程序测试该问题之前未曾发现的。事实上，我们经常通过给系统提供比所需更多的隐藏单元，并观察到它没有使用其中一些，从而发现这些更优雅的解。对实际发现的解进行分析，常常引导我们发现涉及更少隐藏单元的更好解。
will be on. The next most important thing to note about the solution is that the weights on each side of the midpoint of the string are in the ratio of 1 : 2:4. This ensures that each of the eight patterns that can occur on each side of the midpoint sends a unique activation sum to the hidden unit. This assures that there is no pattern on the left that will exactly balance a non-mirror-image pattern on the right. Finally, the two hidden units have identical patterns of weights from the input units except for sign. This ensures that for every nonsymmetric pattern, at least one of the two hidden units will come on and turn on the output unit. To summarize, the network is arranged so that both hidden units will receive exactly zero activation from the input units when the pattern is symmetric, and at least one of them will receive positive input for every nonsymmetric pattern. This problem was interesting to us because the learning system developed a much more elegant solution to the problem than we had previously considered. This problem was not the only one in which this happened. The parity solution discovered by the learning procedure was also one that we had not discovered prior to testing the problem with our learning procedure. Indeed, we frequently discover these more elegant solutions by giving the system more hidden units than it needs and observing that it does not make use of some of those provided. Some analysis of the actual solutions discovered often leads us to the discovery of a better solution involving fewer hidden units.
我们在学习算法上测试的另一个有趣问题是简单的二进制加法问题。这个问题之所以有趣，是因为它有一个非常优雅的解，因为它是我们发现的能够可靠找到局部极小值的问题之一，并且避免这些局部极小值的方法让我们对局部极小值可能被发现和避免的条件有了一些洞察。图 10 展示了基本问题及其最小解。有四个输入单元、三个输出单元和两个隐藏单元。输出模式可以视为由输入模式表示的两个两位二进制数之和的二进制表示。图中的第二和第四个输入单元对应于两个二进制数的低位，第一和第三个单元对应于两个高位。隐藏单元对应于求和中的进位。因此，最右边的隐藏单元在输入模式中的两个低位都为 1 时激活，左边的隐藏单元在两个高位都为 1 时激活。342 基本机制
Another interesting problem on which we have tested our learning algorithm is the simple binary addition problem. This problem is interesting because there is a very elegant solution to it, because it is the one problem we have found where we can reliably find local minima and because the way of avoiding these local minima gives us some insight into the conditions under which local minima may be found and avoided. Figure 10 illustrates the basic problem and a minimal solution to it. There are four input units, three output units, and two hidden units. The output patterns can be viewed as the binary representation of the sum of two two-bit binary numbers represented by the input patterns. The second and fourth input units in the diagram correspond to the low-order bits of the two binary numbers and the first and third units correspond to the two higher order bits. The hidden units correspond to the carry bits in the summation. Thus the hidden unit on the far right comes on when both of the lower order bits in the input pattern are 1, and the one on the left comes on when both of the higher order bits are 1. 342 BASIC MECHANISMS
输出单元 输入单元 隐藏单元
Output Units Input Units Hidden Units
图 10. 用于相加两个两位二进制数的最小网络。有四个输入单元，三个输出单元，两个隐藏单元。输出模式可以被视为由输入模式表示的两个两位二进制数之和的二进制表示。图中的第二个和第四个输入单元对应于两个二进制数的低位，第一和第三个单元对应于两个高位。隐藏单元对应于求和中的进位位。最右边的隐藏单元在输入模式中的两个低位都开启时开启，左边的隐藏单元在两个高位都开启时，或者一个高位与另一个隐藏单元开启时开启。
Figure 10. Minimal network for adding two two-bit binary numbers. There are four input units, three output units, and two hidden units. The output patterns can be viewed as the binary representation of the sum of two two-bit binary numbers represented by the input patterns. The second and fourth input units in the diagram correspond to the low-order bits of the two binary numbers, and the first and third units correspond to the two higher-order bits. The hidden units correspond to the carry bits in the summation. The hidden unit on the far right comes on when both of the lower-order bits in the input pattern are turned on, and the one on the left comes on when both higher-order bits are turned on or when one of the higher-order bits and the other hidden unit is turned on.
所有连接上的权重假设为+1，除非另有说明。负连接
The weights on all lines are assumed to be +1 except where noted. Negative connec
由虚线表示。像往常一样，偏置由圆圈内的数字表示。
tions are indicated by dashed lines. As usual, the biases are indicated by the numbers in the circles.
当两个高位都开启或其中一个
on when both higher order bits are turned on or when one of the
高位和另一个隐藏单元开启时。在图中，所有连接上的权重假设为+1，除非
higher order bits and the other hidden unit is turned on. In the diagram, the weights on all lines are assumed to be +1 except where
注意到，抑制连接由虚线表示。与通常一样，偏置由圆圈中的数字表示。要理解这个网络的工作原理，注意到最低阶输出位由两个低阶输入位之间的异或决定是有用的。解决这个异或问题的一种方法是设置一个隐藏单元
Note that inhibitory connections are indicated by dashed lines. As usual, the biases are indicated by the numbers in the circles. To understand how this network works, it is useful to note that the lowest order output bit is determined by an exclusive-or among the two low-order input bits. One way to solve this XOR problem is to have a hidden unit
当两个低阶输入位都为一时激活，然后让它抑制
come on when both low-order input bits are on and then have it inhibit
输出单元。否则，任一低阶输入单元都可以打开
the output unit. Otherwise either of the low-order input units can turn
低阶输出位。中间位稍微更
on the low-order output bit. The middle bit is somewhat more
版权材料 8. 学习内部表示 343
Copyrighted Material 8. LEARNING INTERNAL REPRESENTATIONS 343
困难。注意，当包含两个高阶输入位和低阶进位位的集合中有奇数个位被开启时，中间位应该被开启。观察将确认所示的网络执行这一任务。最左边的隐藏单元接收来自两个高阶位和进位位的输入。其偏置使得当两个或更多输入被开启时它会被开启。中间输出单元接收来自相同三个单元的正输入，以及来自第二个隐藏单元的-2 的负输入。这确保了当三个输入中恰好一个被开启时，第二个隐藏单元保持关闭，输出位被开启。当三个输入中恰好两个被开启时，隐藏单元会被开启并抵消两个单元对输出位的兴奋输入，所以输出位保持关闭。最后，当三个输入都被开启时，输出位将接收-2
difficult. Note that the middle bit should come on whenever an odd number of the set containing the two higher order input bits and the lower order carry bit is turned on. Observation will confirm that the network shown performs that task. The left-most hidden unit receives inputs from the two higher order bits and from the carry bit. Its bias is such that it will come on whenever two or more of its inputs are turned on. The middle output unit receives positive inputs from the same three units and a negative input of -2 from the second hidden unit. This ensures that whenever just one of the three are turned on, the second hidden unit will remain off and the output bit will come on. Whenever exactly two of the three are on, the hidden unit will turn on and counteract the two units exciting the output bit, so it will stay off. Finally, when all three are turned on, the output bit will receive -2
从它的进位位得到-2，从其他三个输入得到+3。净输入为正，所以中间单元将被开启。最后，当第二个隐藏单元开启时，第三个输出位应被开启——也就是说，每当有来自第二位的进位时。这样我们就有了一个执行当前任务的最小网络。此外，应注意这一网络背后的概念可推广到任意数量的
from its carry bit and +3 from its other three inputs. The net is positive, so the middle unit will be on. Finally, the third output bit should turn on whenever the second hidden unit is on - that is, whenever there is a carry from the second bit. Here then we have a minimal network to carry out the job at hand. Moreover, it should be noted that the concept behind this network is generalizable to an arbitrary number
的输入和输出位。一般来说，对于两个 m 位二进制数相加，我们需要 2m 个输入单元、m 个隐藏单元和 m+1 个输出单元。不幸的是，这是我们发现的一个会可靠地将系统引入局部极小值的问题。在此问题的学习试验开始时，我们允许任意输入单元连接到任意输出单元和任意隐藏单元。我们允许任意隐藏单元连接到任意输出单元，并且我们允许一个隐藏单元连接到另一个隐藏单元，但由于不能有回路，相反方向的连接是不允许的。有时系统会发现在本质上与图中相同的网络。然而，通常系统会陷入局部极小值。当低位上的异或问题没有以图中所示方式解决时，问题就会出现。一种失败的方式是当两个隐藏单元中“较高的”被“选中”来解决异或问题时。这会产生问题，因为另一个隐藏单元无法“看到”进位位，因此最终无法解决该问题。这个问题似乎源于这样一个事实：第二位输出的学习总是依赖于第一位的学习（因为关于进位的信息对学习第二位是必要的），因此落后于第一位的学习，并且对选择隐藏单元没有影响
of input and output bits. In general, for adding two m-bit binary numbers we will require 2m input units, m hidden units, and m+1 output units. Unfortunately, this is the one problem we have found that reliably leads the system into local minima. At the start in our learning trials on this problem we allow any input unit to connect to any output unit and to any hidden unit. We allow any hidden unit to connect to any output unit, and we allow one of the hidden units to connect to the other hidden unit, but, since we can have no loops, the connection in the opposite direction is disallowed. Sometimes the system will discover essentially the same network shown in the figure. Often, however, the system ends up in a local minimum. The problem arises when the XOR problem on the low-order bits is not solved in the way shown in the diagram. One way it can fail is when the "higher" of the two hidden units is "selected" to solve the XOR problem. This is a problem because then the other hidden unit cannot "see" the carry bit and therefore cannot finally solve the problem. This problem seems to stem from the fact that the learning of the second output bit is always dependent on learning the first (because information about the carry is necessary to learn the second bit) and therefore lags behind the learning of the first bit and has no influence on the selection of a hidden unit to
除了最高位之外是相同的。当三个或更多输入单元被开启时，最高位总是被开启。这总是首先被学习到。
is the same except for the highest order bit. The highest order bit is always on whenever three or more of the input units are on. This is always learned first
344 基本机制
344 Basic Mechanisms
解决第一个 XOR 问题。因此，大约有一半的时间（在这个问题中）会选择错误的单元，问题无法解决。在这种情况下，系统能找到除以下和之外的所有解：
Solve the first XOR problem. Thus, about half of the time (in this problem) the wrong unit is chosen and the problem cannot be solved. In this case, the system finds a solution for all of the sums except the
11 + 11 - 110 (3+3=6) 的情况中，它错过了向中间位的进位，得到 11 + 11 - 100。这个问题与我们解决的其他问题不同，因为这里的隐藏单元不是“等势的”。在我们的大多数其他问题中，隐藏单元是等势的，因此没有出现这个问题。不过，应该指出，有一个相对简单的解决方法——即添加一些额外的隐藏单元。在这种情况下，我们可以承受在一个或多个选择上犯错，系统仍然可以解决问题。对于两位数的加法问题，我们发现系统总是通过一个额外的隐藏单元就能解决问题。对于更大的数字，可能需要两到三个。为了说明，我们展示了一次运行的结果，该运行使用了三个隐藏单元，而不是最小的两个。图 11
11 + 11 - 110 (3+3=6) case in which it misses the carry into the middle bit and gets 11 + 11 - 100 instead. This problem differs from others we have solved in as much as the hidden units are not "equipotential" here. In most of our other problems the hidden units have been equipotential, and this problem has not arisen. It should be noted, however, that there is a relatively simple way out of the problem - namely, add some extra hidden units. In this case we can afford to make a mistake on one or more selections and the system can still solve the problems. For the problem of adding two-bit numbers we have found that the system always solves the problem with one extra hidden unit. With larger numbers it may require two or three more. For purposes of illustration, we show the results of one of our runs with three rather than the minimum two hidden units. Figure 11
展示了网络在每种输入模式呈现 3020 次后，以学习率\(η = 0.5\)达到的状态。为方便起见，我们将网络分成四个部分展示。在图 11A 中，我们展示了与隐藏单元的连接以及隐藏单元之间的连接。该图显示了为该问题生成的内部表示。“最低”隐藏单元在任意低阶位开启时关闭。换句话说，它检测到没有低阶位开启的情况。“最高”隐藏单元的配置是，当和小于 2 时开启。中间隐藏单元开启的条件更为复杂。表 8 显示了每个十六种输入模式对应的隐藏单元模式。图 11B
shows the state reached by the network after 3,020 presentations of each input pattern and with a learning rate of \(η = 0.5\). For convenience, we show the network in four parts. In Figure 11A we show the connections to and among the hidden units. This figure shows the internal representation generated for this problem. The "lowest" hidden unit turns off whenever either of the low-order bits are on. In other words it detects the case in which no low-order bit is turn on. The "highest" hidden unit is arranged so that it comes on whenever the sum is less than two. The conditions under which the middle hidden unit comes on are more complex. Table 8 shows the patterns of hidden units which occur to each of the sixteen input patterns. Figure 11B
sol ve the fi rst XOR probl em . Th u s , about half of the ti me ( i n this problem) t h e wrong unit i s chosen and the problem cannot be sol ved . In t h i s case , t h e syste m finds a sol u t i on for all of the sums except the
输出单元 8. 学习内部表示 345
Output Units 8. LEARNING INTERNAL REPRESENTATIONS 345
公式片段，无法准确还原，请检查原始文档
\/1\\, 1 1 1\ \ 1 \ \ I I \ \ \I 1.,1 \ \
图 11. 为求和问题找到的网络。A: 来自
FIGURE 11. Network found for the summation problem. A: The connections from the
输入单元到三个隐藏单元以及隐藏单元之间的连接。B:
input units to the three hidden units and the connections among the hidden units. B:
从输入和隐藏单元到最低阶输出单元的连接。C:
The connections from the input and hidden units to the lowest order output unit. C: The
从输入和隐藏单元到中间输出单元的连接。D: 从输入和隐藏单元到最高阶输出单元的连接。
connections from the input and hidden units to the middle output unit. D: The connections from the input and hidden units to the highest
版权材料 346 基本机制
Copyrighted Material 346 BASIC MECHANISMS
1100010111+0001001111+0100010011+1000010111+11000110
1100010111+0001001111+0100010011+1000010111+11000110
内部表示是分布式的，重要的是隐藏单元上的活动模式，而不是任何特定隐藏单元的含义。
the internal representations are distributed and it is the pattern of activity over the hidden units, not the meaning of any particular hidden unit that is important.
考虑这样一种情况：系统输入由 n+1 个二进制值的模式组成，输出为 n 个值。进一步假设一般规则是，n 个输入单元应直接映射到输出模式。然而，其中一个输入位是特殊的，它是一个取反位。当该位关闭时，模式的其他部分应该直接映射；但当它开启时，应该映射模式的补码到输出。表 9 显示了相应的映射。在这种情况下，输入模式的最左边元素是取反位，但系统无法知道这一点，必须学习哪个位是取反位。在这种情况下，允许从任何输入单元到任何隐藏或输出单元，以及从任何隐藏单元到任何输出单元的权重。系统学会了将所有权重设置为零，除了图 12 中所示的那些。问题和解决方案的基本结构在图中有明显的体现。显然，问题被简化为取反位与三个异或（XOR）之间的一组运算。
Consider a situation in which the input to a system consists of patterns of n + 1 binary values and an output of n values. Suppose further that the general rule is that n of the input units should be mapped directly to the output patterns. One of the input bits, however, is special. It is a negation bit. When that bit is off, the rest of the pattern is supposed to map straight through, but when it is on, the complement of the pattern is to be mapped to the output. Table 9 shows the appropriate mapping. In this case the left element of the input pattern is the negation bit, but the system has no way of knowing this and must learn which bit is the negation bit. In this case, weights were allowed from any input unit to any hidden or output unit and from any hidden unit to any output unit. The system learned to set all of the weights to zero except those shown in Figure 12. The basic structure of the problem and of the solution is evident in the figure. Clearly the problem was reduced to a set of three XORs between the negation bit
版权资料 8. 学习内部表征 347
Copyrighted Material 8. LEARNING INTERNAL REPRESENTATIONS 347
表 9 输入模式 输出模式
TABLE 9 Input Patterns Output Patterns
00000001001000110100010101100111100010011010
00000001001000110100010101100111100010011010
并且每个输入。在两个最右侧输入单元的情况下，异或问题通过招募一个隐藏单元来解决，该隐藏单元检测否定单元和对应输入单元均未开启的情况。在第三种情况下，隐藏单元检测否定单元和相关输入都开启的情况。在这种情况下，问题在少于 5,000 次遍历刺激集时得到解决，其中⟀(⟀eta = 0.25⟀)。
and each input. In the case of the two rightmost input units, the XOR problems were solved by recruiting a hidden unit to detect the case in which neither the negation unit nor the corresponding input unit was on. In the third case, the hidden unit detects the case in which both the negation unit and relevant input were on. In this case the problem was solved in less than 5,000 passes through the stimulus set with ⟀(⟀eta = 0.25⟀).
图 12.
Figure 12.
图 12. 针对否定问题发现的解决方案。最左边的单元是
Figure 12. The solution discovered for the negation problem. The left-most unit is
否定单元。该问题已被简化为三个异或问题，
the negation unit. The problem has been reduced and solved as three exclusive-ors
涉及否定单元与另外三个单元中的每一个。
between the negation unit and each of the other three units.
版权资料 348 基本机制
Copyrighted Material 348 BASIC MECHANISMS
到目前为止讨论的大多数问题（除了对称性问题）都是相当抽象的数学问题。现在我们转向一个更具几何性的问题——区分 T 和 C，且不受平移和旋转的影响。图 13 显示了这些实验中使用的刺激模式。注意，这些模式由
Most of the problems discussed so far (except the symmetry problem) are rather abstract mathematical problems. We now turn to a more geometric problem - that of discriminating between a T and a C - independent of translation and rotation. Figure 13 shows the stimulus patterns used in these experiments. Note, these patterns are
五个方块组成，彼此之间只差一个方块。此外，正如 Minsky 和 Papert（1969）指出的，当考虑所有可能的平移和旋转下的模式集合时，
each made of five squares and differ from one another by a single square. Moreover, as Minsky and Papert (1969) point out, when considering the set of patterns over all possible translations and rotations
（90°、180°和 270°旋转），这些模式在方块对之间的距离集合上并无差异。要看到模式集之间的差异，至少需要观察三方块构型。因此 Minsky 和 Papert 称之为三阶问题。5
(of 90°, 180°, and 270°), the patterns do not differ in the set of distances among their pairs of squares. To see a difference between the sets of patterns one must look, at least, at configurations of triplets of squares. Thus Minsky and Papert call this a problem of order three. 5
为了便于学习，针对这个问题采用了一种相当不同的架构。图 14 显示了所使用网络的基本结构。输入模式现在被概念化为叠加在矩形网格上的二维模式。不是允许每个输入单元连接到每个隐藏单元，而是隐藏单元本身被组织成一个二维网格，每个单元接收来自输入空间一个 3×3 方形区域的输入。在这个意义上，重叠的方形区域构成了隐藏单元的预定义
In order to facilitate the learning, a rather different architecture was employed for this problem. Figure 14 shows the basic structure of the network we employed. Input patterns were now conceptualized as two dimensional patterns superimposed on a rectangular grid. Rather than allowing each input unit to connect to each hidden unit, the hidden units themselves were organized into a two-dimensional grid with each unit receiving input from a square 3 x 3 region of the input space. In this sense, the overlapping square regions constitute the predefined
感受野。每个隐藏单元，在整个视野内，馈入一个输出单元，该输出单元应取
receptive field of the hidden units. Each of the hidden units, over the entire field, feeds into a single output unit which is to take on the value
版权所有材料 8. 学习内部表示 349
Copyrighted Material 8. LEARNING INTERNAL REPRESENTATIONS 349
如果输入是 T（在任何位置或方向上）则输出为 1，如果输入是 C 则输出为 0。此外，为了使发生的学习独立于模式出现在场上的位置，我们约束所有单元学习完全相同的权重模式。通过这种方式，每个单元被约束在其感受野上计算完全相同的函数——感受野被约束为具有相同的形状。这保证了平移独立性，并避免了学习中任何可能的“边缘效应”。这种学习可以很容易地扩展到任意大的输入单元场。这个约束是通过简单地将每个单元由 Delta 规则决定的权重变化相加，然后以完全相同的量更改所有权重来实现的。
1 if the input is a T (at any location or orientation) and 0 if the input is a C. Further, in order that the learning that occurred be independent of where on the field the pattern appeared, we constrained all of the units to learn exactly the same pattern of weights. In this way each unit was constrained to compute exactly the same function over its receptive field - the receptive fields were constrained to all have the same shape. This guarantees translation independence and avoids any possible "edge effects" in the learning. The learning can readily be extended to arbitrarily large fields of input units. This constraint was accomplished by simply adding together the weight changes dictated by the delta rule for each unit and then changing all weights the exact same amount. In
通过这种方式，整个隐藏单元场仅仅是一个位于输入空间不同区域的单一特征检测器的复制，并且在场的一个部分发生的学习自动地泛化到场中的其他部分。 6
this way, the whole field of hidden units consists simply of replications of a single feature detector centered on different regions of the input space, and the learning that occurs in one part of the field is automatically generalized to the rest of the field. 6
我们以这种方式多次运行这个问题。结果，我们找到了许多解决方案。理解该系统的最简单方法或许是查看隐藏单元感受野的形式。图 15 展示了我们看到的几个感受野。7 图 15A 显示了所发展的最局部表示。这种中心-周边-抑制检测器被证明是一种极好的 T 检测器。因为，如图所示，T 可以延伸到中心并获得+1 的净输入，所以该检测器将对任何方向的 T 被激活。另一方面，任何延伸到中心的 C 必须覆盖至少两个抑制性细胞。使用这个检测器，可以设置偏置，使得每当呈现 T 时，整个抑制性单元场中只有一个会被激活，而任何 C 都不会激活任何隐藏单元。这是一种凸起检测器，通过检测 T 的凸起来区分 T 和 C。
We have run this problem in this way a number of times. As a result, we have found a number of solutions. Perhaps the simplest way to understand the system is by looking at the form of the receptive field for the hidden units. Figure 15 shows several of the receptive fields we have seen. 7 Figure 15A shows the most local representation developed. This on-center-off-surround detector turns out to be an excellent T detector. Since, as illustrated, a T can extend into the on center and achieve a net input of +1, this detector will be turned on for a T at any orientation. On the other hand, any C extending into the center must cover at least two inhibitory cells. With this detector the bias can be set so that only one of the whole field of inhibitory units will come on whenever a T is presented and none of the hidden units will be turned on by any C. This is a kind of protrusion detector which differentiates between a T and C by detecting the protrusion of the T.
图 15B 所示的感受野又是一种 T 检测器。每个 T 以一个+2 的量激活一个隐藏单元，并且任何隐藏单元从任何 C 处获得的输入不超过+1。如图所示，90°和 270°方向的 T 向横杆对齐的隐藏单元发送总计+2。另外两个方向的 T 通过检测这两个字符的垂直凸起的方式接收+2。图 15C 显示了一种更分布的表示。如图所示，每个 T 激活五个不同的隐藏单元，而每个 C 仅激发三个隐藏单元。在这种情况下，系统再次基于 T 的凸起末端（C 没有该末端）来区分字符。最后，图 15D 所示的感受野更加有趣。在这种情况下，每个隐藏单元具有正偏置，因此它默认是开启的，除非被关闭。抑制权重的强度使得如果一个字符与隐藏单元的感受野重叠，该单元就会关闭。该系统之所以有效，是因为 C 比 T 更紧密，因此 T 关闭的单元比 C 多。T 关闭了 21 个
The receptive field shown in Figure 15B is again a kind of T detector. Every T activates one of the hidden units by an amount +2 and none of the hidden units receives more than +1 from any of the C's. As shown in the figure, T's at 90° and 270° send a total of +2 to the hidden units on which the crossbar lines up. The T's at the other two orientations receive +2 from the way it detects the vertical protrusions of those two characters. Figure 15C shows a more distributed representation. As illustrated in the figure, each T activates five different hidden units whereas each C excites only three hidden units. In this case the system again is differentiating between the characters on the basis of the protruding end of the T which is not shared by the C. Finally, the receptive field shown in Figure 15D is even more interesting. In this case every hidden unit has a positive bias so that it is on unless turned off. The strength of the inhibitory weights are such that if a character overlaps the receptive field of a hidden unit, that unit turns off. The system works because a C is more compact than a T and therefore the T turns off more units than the C. The T turns off 21
隐藏单元，并且 C 只关闭了 20 个。这是一个真正的分布式
hidden units, and the C turns off only 20. This is a truly distributed
8. 学习内部表示 3.5.1
8. Learning Internal Representations 3.5.1
图 15. 在 T-C 问题的不同运行中发现的感受野。A：用于检测 T 的非中心-周围抑制感受野。B：一个垂直条形检测器，它对 T 的响应强于 C。C：一个对角条形检测器。一个 T 激活五个这样的检测器，而一个 C 仅激活三个这样的检测器。D：一个紧凑度检测器。这个抑制性感受野在输入覆盖其感受野的任何区域时关闭。由于 C 比 T 更紧凑，它关闭了 20 个这样的检测器
FIGURE 15. Receptive fields found in different runs of the T-C problem. A: A non center-off-surround receptive field for detecting T's. B: A vertical bar detector which responds to T's more strongly than C's. C: A diagonal bar detector. A T activates five such detectors whereas a C activates only three such detectors. D: A compactness detector. This inhibitory receptive field turns off whenever an input covers any region of its receptive field. Since the C is more compact than the T it turns off 20 such detectors
而 T 关闭了其中的 21 个。
whereas the T turns off 21 of them.
表示。在每种情况下，解决方案在大约 5,000 到 10,000 次呈现八个图案集合时达到。8
representation. In each case, the solution was reached in from about 5,000 to 10,000 presentations of the set of eight patterns. 8
有趣的是，图 15D 中所示的抑制性感受野类型是最常见的，并且在这个以及我们所有的模拟中，抑制性连接占主导地位。这可以通过考虑学习通常移动的轨迹来理解。首先，当系统呈现一个
It is interesting that the inhibitory type of receptive field shown in Figure 15D was the most common and that there is a predominance of inhibitory connections in this and indeed all of our simulations. This can be understood by considering the trajectory through which the learning typically moves. At first, when the system is presented with a
8 由于平移不变性被内置到学习过程中，输入发生在哪里没有区别：无论图案出现在哪里，都会学到同样的东西。因此，只有 5 个图案需要呈现给系统。352 基本机制
8 Since translation independence was built into the learning procedure, it makes no difference where the input occurs: the same thing will be learned wherever the pattern is presented. Thus, there are only 5 patterns to be presented to the system. 352 BASIC MECHANISMS
在困难问题中，初始随机连接误导的可能性与给出正确答案的可能性一样大。在这种情况下，输出单元取值为 0.5 比取更极端值更好。这源于方程 2 中给出的误差函数形式。
In a difficult problem, the initial random connections are as likely to mislead as to give the correct answer. In this case, it is best for the output units to take on a value of 0.5 than to take on a more extreme value. This follows from the form of the error function given in Equation 2.
输出单元可以通过关闭馈入它的单元来实现 0.5 的恒定输出。因此，在几乎每一个困难问题中，首先发生的是隐藏单元被关闭。实现这一点的一种方式是让输入单元抑制隐藏单元。随着系统开始理清头绪并学习适当的函数，一些连接通常会变为正，但大多数连接仍保持负。这种偏向于涉及抑制性输入的解决方案往往会导致非直观的结果，即隐藏单元通常处于开启状态，除非被输入关闭。
The output unit can achieve a constant output of 0.5 by turning off those units feeding into it. Thus, the first thing that happens in virtually every difficult problem is that the hidden units are turned off. One way to achieve this is to have the input units inhibit the hidden units. As the system begins to sort things out and to learn the appropriate function, some of the connections will typically go positive, but the majority of the connections will remain negative. This bias for solutions involving inhibitory inputs can often lead to nonintuitive results in which hidden units are often on unless turned off by the input.
在本节中，我们展示了部分结果。除了在讨论的问题上研究我们的学习系统外，我们还使用了反向传播来学习二进制乘法
We have offered a sample of our results in this section. In addition to having studied our learning system on the problems discussed here, we have employed back propagation for learning to multiply binary
数字、玩井字棋、区分垂直线和水平线、执行一系列动作、识别字符、关联随机向量以及其他众多应用。在所有这些应用中，我们发现广义 delta 规则能够生成所需问题的内部表示类型。我们发现局部极小值非常罕见，而
digits, to play tic-tac-toe, to distinguish between vertical and horizontal lines, to perform sequences of actions, to recognize characters, to associate random vectors, and a host of other applications. In all of these applications we have found that the generalized delta rule was capable of generating the kinds of internal representations required for the problems in question. We have found local minima to be very rare and
系统在合理的时间段内学习。仍然需要更多此类研究来精确理解系统在何种条件下会受局部极小值困扰。可以说，到目前为止该问题并不严重。我们现在转向
that the system learns in a reasonable period of time. Still more studies of this type will be required to understand precisely the conditions under which the system will be plagued by local minima. Suffice it to say that the problem has not been serious to date. We now turn to a
我们深入研究了前馈网络和半线性激活函数上广义 delta 规则的学习特性。有趣的是，这些并非学习过程所能适用的最一般情况。迄今为止，我们仅研究了更完全广义系统的一些示例，但将相同学习规则应用于 sigma-pi 单元和循环网络相对容易。我们在此简要说明。8. 学习内部表示 353 广义 Delta 规则与 Sigma-Pi 单元
We have intensively studied the learning characteristics of the generalized delta rule on feedforward networks and semilinear activation functions. Interestingly, these are not the most general cases to which the learning procedure is applicable. As yet we have only studied a few examples of the more fully generalized system, but it is relatively easy to apply the same learning rule to sigma-pi units and to recurrent networks. We will simply state here. 8. LEARNING INTERNAL REPRESENTATIONS 353 The Generalized Delta Rule and Sigma-Pi Units
读者应回想起第二章，对于 sigma-pi 单元，我们有
It will be recalled from Chapter 2 that in the case of sigma-pi units we have
其中 i 遍历输入到单元 j 的合取集，k 遍历合取的元素。为简化阐述，我们限定在合取不包含超过两个元素的情形。此时，我们可以将单元 i 和 j 的合取到单元 k 的权重记为\(w_{kij}\)。因此，从单元 i 到单元 j 的直接连接上的权重为\(w_{ji}\)，且由于关系是乘性的，\(w_{kij} = w_{kji}\)。现在我们可以将方程 17 重写为
where i varies over the set of conjuncts feeding into unit j and k varies over the elements of the conjuncts. For simplicity of exposition, we restrict ourselves to the case in which no conjuncts involve more than two elements. In this case we can notate the weight from the conjunction of units i and j to unit k by \(w_{kij}\). The weight on the direct connection from unit i to unit j would, thus, be \(w_{ji}\), and since the relation is multiplicative, \(w_{kij} = w_{kji}\). We can now rewrite Equation 17 as
我们现在设定。求导并化简，我们得到 sigma-pi 单元的一个规则，该规则严格类似于半线性激活函数的规则：
We now set Taking the derivative and simplifying, we get a rule for sigma-pi units strictly analogous to the rule for semilinear activation functions:
通过观察图 16，我们可以看到这种情况下误差信号\(\delta\)的正确形式。考虑单元\(U_i\)的适当\(\delta\)值。
We can see the correct form of the error signal, \(\delta\), for this case by inspecting Figure 16. Consider the appropriate value of \(\delta\) for unit \(U_i\).
如图所示。与之前一样，\delta_i\的正确值由所有\U_i\馈入的单元的\delta\之和给出，加权因子为\U_i\的激活乘以激活函数的导数。对于半线性函数，一个单元对另一个单元的影响程度简单地由连接第一个单元与第二个单元的权重\W\给出。在此情况下，\U_i\对\U_k\的影响
In the figure. As before, the correct value of \delta_i\ is given by the sum of the \delta\'s for all of the units into which \U_i\ feeds, weighted by the amount of effect due to the activation of \U_i\ times the derivative of the activation function. In the case of semilinear functions, the measure of a unit's effect on another unit is given simply by the weight \W\ connecting the first unit to the second. In this case, the \U_i\'s effect on \U_k\
不仅取决于\W_{k,i}\，还取决于\U_i\的值。因此，我们有 \(delta_i = f'(net_i) \sum_k \delta_k W_{k,i} O_i\)\
depends not only on \W_{k,i}\, but also on the value of \U_i\. Thus, we have \(delta_i = f'(net_i) \sum_k \delta_k W_{k,i} O_i\)\
如果\U_i\不是输出单元，则与之前一样，\delta_i = f'(net_i) (t_i - o_i)\
if \U_i\ is not an output unit and, as before, \delta_i = f'(net_i) (t_i - o_i)\
如果它是输出单元。版权所有 354 基本机制
if it is an output unit. Copyrighted Material 354 BASIC MECHANISMS
到目前为止，我们一直局限于前馈网络。这看起来是一个很大的限制，但正如 Minsky 和 Papert 所指出的，对于每一个循环网络，都存在一个行为相同的前馈网络（在有限时间内）。我们现在将说明如何构建这一结构，并进而展示循环网络学习规则的正确形式。考虑图 17A 中所示的简单循环网络。图 17B 显示了同一网络的前馈架构。循环网络的行为可以在前馈网络中实现，代价是多次复制硬件以得到前馈版本。9 我们在每个时间点都有不同的单元和不同的权重。为了方便命名，我们对每个单元加上下标，表示对应循环网络中的单元编号以及它所代表的时间。只要我们约束前馈网络每一层的权重相同，我们就得到了一个与图 17A 中循环网络行为完全相同的前馈网络。
We have thus far restricted ourselves to feedforward nets. This may seem like a substantial restriction, but as Minsky and Papert point out, there is, for every recurrent network, a feedforward network with identical behavior (over a finite period of time). We will now indicate how this construction can proceed and thereby show the correct form of the learning rule for the recurrent network. Consider the simple recurrent network shown in Figure 17A. The same network in a feedforward architecture is shown in Figure 17B. The behavior of a recurrent network can be achieved in a feedforward network at the cost of duplicating the hardware many times over for the feedforward version of the network. 9 We have distinct units and distinct weights for each point in time. For naming convenience, we subscript each unit with its unit number in the corresponding recurrent network and the time it represents. As long as we constrain the weights at each level of the feedforward network to be the same, we have a feedforward network which performs identically with the recurrent network of Figure 17A.
图 17 展示了行为相同的循环网络和前馈网络的对比。A：一个全连接的循环网络，包含两个单元。B：一个
Figure 17, A comparison of a recurrent network and a feedforward network with identical behavior. A: A completely connected recurrent network with two units. B: A
前馈网络，其行为与循环网络相同。在这种情况下，每个时间步都有一个单独的单元，并且要求连接每个
feedforward network which behaves the same as the recurrent network. In this case, we have a separate unit for each time step and we require that the weights connecting each
单元层到下一层的权重在所有层中都相同。此外，这些权重必须与循环情况下的对应权重相同。
layer of units to the next be the same for all layers. Moreover, they must be the same as the analogous weights in the recurrent case.
维持所有权重相等这一约束的恰当方法是，简单地跟踪每个层级上为每个权重指定的变化，然后根据这些变化的总和来改变每个权重。
The appropriate method for maintaining the constraint that all weights be equal is simply to keep track of the changes dictated for each weight at each level and then change each of the weights according to the sum
这些单独指定的变化之和。现在，一般规则是
of these individually prescribed changes. Now, the general rule for
确定系统中某个权重在特定时间所需的变化，就是适当误差的加权和。
Determining the change prescribed for a weight in the system for a particular time is simply the weighted sum of an appropriate error.
测量δ和输入沿着相关的线在适当的时间。因此，为循环网络指定正确学习规则的问题，就是确定每个时间δ的适当值。在前馈网络中，我们通过将激活函数的导数乘以它所馈入的那些单元的δ乘上连接强度之和来确定δ。相同的过程适用于循环网络——但在此情况下，与特定单元相关联的δ值会随时间变化，因为单元将误差向后传递，有时传递给自己。每次迭代后，当误差通过网络向后传播时，该次迭代的权重变化必须添加到之前迭代指定的权重变化中，并存储总和。这种通过网络传递误差的过程应持续与原始激活传播相同的迭代次数。此时，可以对所有权重进行适当的更改。通常，循环网络的程序是：向系统呈现一个输入（通常是一个序列），同时系统运行若干次迭代。在系统运行的某些指定时间，将某些单元的输出与该单元在该时间的目标进行比较，并生成误差信号。然后，每个这样的误差信号被向后传递通过网络，传递次数等于前向传递中使用的迭代次数。在每次迭代中计算权重变化，并保存针对特定权重的所有权重变化的总和。最后，在所有这样的误差信号通过网络传播后，更改权重。这个过程的主要问题是所需的内存。系统不仅需要在误差传播时保持其权重变化总和，而且每个单元必须以某种方式记录其在原始处理过程中被驱动的激活值序列。这是因为在每次迭代中，当误差通过网络向后传递时，当前δ对应于更早的时间点，所需的权重变化依赖于当时单元的激活水平。目前尚不完全清楚这种机制如何在大脑中实现。
Measure δ and the input along the relevant line both for the appropriate times. Thus, the problem of specifying the correct learning rule for recurrent networks is simply one of determining the appropriate value of δ for each time. In a feedforward network we determine δ by multiplying the derivative of the activation function by the sum of the δ's for those units it feeds into weighted by the connection strengths. The same process works for the recurrent network - except in this case, the value of δ associated with a particular unit changes in time as a unit passes error back, sometimes to itself. After each iteration, as error is being passed back through the network, the change in weight for that iteration must be added to the weight changes specified by the preceding iterations and the sum stored. This process of passing error through the network should continue for a number of iterations equal to the number of iterations through which the activation was originally passed. At this point, the appropriate changes to all of the weights can be made. In general, the procedure for a recurrent network is that an input (generally a sequence) is presented to the system while it runs for some number of iterations. At certain specified times during the operation of the system, the output of certain units are compared to the target for that unit at that time and error signals are generated. Each such error signal is then passed back through the network for a number of iterations equal to the number of iterations used in the forward pass. Weight changes are computed at each iteration and a sum of all the weight changes dictated for a particular weight is saved. Finally, after all such error signals have been propagated through the system, the weights are changed. The major problem with this procedure is the memory required. Not only does the system have to hold its summed weight changes while the error is being propagated, but each unit must somehow record the sequence of activation values through which it was driven during the original processing. This follows from the fact that during each iteration while the error is passed back through the system, the current δ is relevant to a point earlier in time and the required weight changes depend on the activation levels of the units at that time. It is not entirely clear how such a mechanism could be implemented in
大脑。然而，令人兴奋的是，这个过程可能非常强大，因为它试图解决的问题相当于找到一个顺序程序（类似于数字计算机的程序），产生指定的输入序列/输出序列对。此外，教师与系统的交互可以非常灵活，例如，如果系统陷入局部最小值，教师可以以“提示”的形式引入
the brain. Nevertheless, it is tantalizing to realize that such a procedure is potentially very powerful, since the problem it is attempting to solve amounts to that of finding a sequential program (like that for a digital computer) that produces specified input-sequence/output-sequence pairs. Furthermore, the interaction of the teacher with the system can be quite flexible, so that, for example, should the system get stuck in a local minimum, the teacher could introduce "hints" in the form of
处理中间阶段的期望输出值。我们在循环网络问题上的经验中，我们进行了一些 [8. 学习内部表示 357]。
desired output values for intermediate stages of processing. Our experience with recurrent net problems we have carried out some 8. Learning Internal Representations 357
实验。我们首先转向一个非常简单的问题，其中系统被诱导发明一个移位寄存器来解决该问题。
experiments. We turn first to a very simple problem in which the system is induced to invent a shift register to solve the problem.
学习成为移位寄存器。也许我们研究的循环问题中最简单的一类是输入和输出单元相同且没有隐藏单元的情况。我们简单地呈现一个模式，让系统处理一段时间。然后系统的状态与某个目标状态进行比较。如果在指定时间未达到目标状态，则向系统注入误差，并修改其权重。然后显示一个新的输入模式并重新开始。在这些情况下，系统内的连接没有约束。任何单元可以连接到任何其他单元。我们研究过的最简单的问题就是所谓的移位寄存器问题。在这个问题中，单元被概念化为一个循环移位寄存器。首先在单元上建立一个任意位模式。然后允许它们处理两个时间步。这两个时间步之后的目标状态是原始模式向左移动两个位置。这里有趣的问题涉及起始状态呈现与目标状态呈现之间单元的状态。问题的一个解决方案是让系统成为一个移位寄存器，并在每个时间段内将模式恰好向左移动一个单元。如果系统这样做，那么两个时间单位后它肯定向左移动了两个位置。我们尝试了三个或五个单元组的问题，并且如果我们约束所有单元的偏置为负（因此单元除非被打开否则关闭），系统总是学会成为这种移位寄存器。因此，尽管原则上任何单元都可以连接到任何其他单元，但系统实际上学会了将所有权重设置为零，除了连接单元与其左邻单元的权重。由于目标状态是在循环寄存器的假设下确定的，最左边的单元发展出了与最右边单元的强连接。系统学习得相对较快。当η=0.25 时，它能在少于 200 次遍历所有可能模式集合内完美学习，无论是三单元还是五单元系统。我们迄今为止描述的任务异常简单，但它们确实说明了算法如何在无约束网络中工作。我们还尝试了循环网络的一些更困难的问题。
Learning to be a shift register. Perhaps the simplest class of recurrent problems we have studied is one in which the input and output units are one and the same and there are no hidden units. We simply present a pattern and let the system process it for a period of time. The state of the system is then compared to some target state. If it hasn't reached the target state at the designated time, error is injected into the system and it modifies its weights. Then it is shown a new input pattern and restarted. In these cases, there is no constraint on the connections in the system. Any unit can connect to any other unit. The simplest such problem we have studied is what we call the shift register problem. In this problem, the units are conceptualized as a circular shift register. An arbitrary bit pattern is first established on the units. They are then allowed to process for two time-steps. The target state, after those two time-steps, is the original pattern shifted two spaces to the left. The interesting question here concerns the state of the units between the presentation of the start state and the time at which the target state is presented. One solution to the problem is for the system to become a shift register and shift the pattern exactly one unit to the left during each time period. If the system did this then it would surely be shifted two places to the left after two time units. We have tried this problem with groups of three or five units and, if we constrain the biases on all of the units to be negative (so the units are off unless turned on), the system always learns to be a shift register of this sort. Thus, even though in principle any unit can connect to any other unit, the system actually learns to set all weights to zero except the ones connecting a unit to its left neighbor. Since the target states were determined on the assumption of a circular register, the left-most unit developed a strong connection to the right-most unit. The system learns this relatively quickly. With η = 0.25 it learns perfectly in fewer than 200 sweeps through the set of possible patterns with either three or five-unit systems. The tasks we have described so far are exceptionally simple, but they do illustrate how the algorithm works with unrestricted networks. We have attempted a few more difficult problems with recurrent networks.
版权所有 358 基本机制
Copyrighted Material 358 BASIC MECHANISMS
其中一个更有趣的问题涉及学习完成模式序列。我们的最后一个例子来自这个领域。
One of the more interesting involves learning to complete sequences of patterns. Our final example comes from this domain.
学习完成序列。表 10 展示了一组 25 个序列，这些序列的选择使得序列的前两项唯一地决定了剩余四项。我们使用这组序列来测试循环网络的学习能力。该网络由五个输入单元(A、B、C、D、E)、30 个隐藏单元和三个输出单元(1、2、3)组成。在时间 1，打开序列第一个项目对应的输入单元，其他输入单元关闭。在时间 2，打开序列第二个项目对应的输入单元，其他所有输入单元关闭。然后所有输入单元关闭，并在前向迭代的剩余四个步骤中保持关闭。网络必须学会使输出单元呈现代表序列其余部分的状态。与简单的前馈网络(或其迭代等价物)不同，误差不仅在最终层或最终时间点进行评估。输出单元必须在正向迭代期间呈现适当的状态，因此在反向传播阶段，通过将记忆的输出单元实际状态与其期望状态进行比较，在每个时间步注入误差。循环网络的学习过程对允许的连接结构没有限制。11 对于序列完成问题，我们使用了从输入单元到隐藏单元以及从隐藏单元到输出单元的单向连接。每个隐藏单元都有到其他每个隐藏单元以及到自身的单向连接。
Learning to complete sequences. Table 10 shows a set of 25 sequences which were chosen so that the first two items of a sequence uniquely determine the remaining four. We used this set of sequences to test out the learning abilities of a recurrent network. The network consisted of five input units (A, B, C, D, E), 30 hidden units, and three output units (1, 2, 3). At Time 1, the input unit corresponding to the first item of the sequence is turned on and the other input units are turned off. At Time 2, the input unit for the second item in the sequence is turned on and the others are all turned off. Then all the input units are turned off and kept off for the remaining four steps of the forward iteration. The network must learn to make the output units adopt states that represent the rest of the sequence. Unlike simple feedforward networks (or their iterative equivalents), the errors are not only assessed at the final layer or time. The output units must adopt the appropriate states during the forward iteration, and so during the back-propagation phase, errors are injected at each time-step by comparing the remembered actual states of the output units with their desired states. The learning procedure for recurrent nets places no constraints on the allowable connectivity structure. 11 For the sequence completion problem, we used one-way connections from the input units to the hidden units and from the hidden units to the output units. Every hidden unit had a one-way connection to every other hidden unit and to itself,
表 10 25 个待学习的序列 AA 1 2 1 2 AB 1 2 2 3 AC 1 2 3 1 AD 1 2 2 1 AE 1 2 1 3 BA 2 3 1 2 BB 2 3 2 3 BC 2 3 3 1 BD 2 3 2 1 BE 2 3 1 3 CA 3 1 1 2 CB 3 1 2 3 CC 3 1 3 1 CD 3 1 2 1 CE 3 1 1 3 DA 2 1 1 2 DB 2 1 2 3 DC 2 1 3 1 DD 2 1 2 1 DE 2 1 1 3 EA 1 3 1 2 EB 1 3 2 3 EC 1 3 3 1 ED 1 3 2 1 EE 1 3 1 3
TABLE 10 25 SEQUENCES TO BE LEARNED AA 1 2 1 2 AB 1 2 2 3 AC 1 2 3 1 AD 1 2 2 1 AE 1 2 1 3 BA 2 3 1 2 BB 2 3 2 3 BC 2 3 3 1 BD 2 3 2 1 BE 2 3 1 3 CA 3 1 1 2 CB 3 1 2 3 CC 3 1 3 1 CD 3 1 2 1 CE 3 1 1 3 DA 2 1 1 2 DB 2 1 2 3 DC 2 1 3 1 DD 2 1 2 1 DE 2 1 1 3 EA 1 3 1 2 EB 1 3 2 3 EC 1 3 3 1 ED 1 3 2 1 EE 1 3 1 3
版权材料。8. 学习内部表示 359 每个输出单元也连接到所有其他输出单元及自身。所有连接以均匀分布在 -0.3 和 +0.3 之间的小随机权重开始。每个序列开始时，所有隐藏单元和输出单元的激活水平均为 0.2。我们使用的学习过程版本中，在改变权重之前，先针对整个示例集计算每个权重的误差梯度。这意味着每个连接必须累积所有示例以及每个示例中涉及的所有时间步的梯度之和。训练期间，我们使用了一组特定的 20 个示例，在几乎完美学习这些示例后，我们在剩余示例上测试网络，看它是否捕捉到了将序列前两项与后四项联系起来的明显规律。结果如表 11 所示。对于五个测试序列中的四个，输出单元在所有时刻都具有正确的值（假设我们将高于 0.5 的值视为 1，低于 0.5 的值视为 0）。网络显然捕捉到了规则：序列的第一项决定第三和第四项，第二项决定第五和第六项。我们用一组不同的随机初始权重重复了模拟，结果所有五个测试序列都正确。学习需要遍历所有 20 个训练序列 260 次。输出单元中的误差计算如下：对于应为开启的单元，如果其激活水平高于 0.8，则误差为零，否则误差的导数为低于 0.8 的量。类似地，对于应关闭的输出单元，误差的导数为高于 0.2 的量。每次遍历后，每个权重减少 0.02 乘以该次遍历累积的总梯度再加上 0.9
Copyrighted Material 8. LEARNING INTERNAL REPRESENTATIONS 359 and every output unit was also connected to every other output unit and to itself. All the connections started with small random weights uniformly distributed between -0.3 and +0.3. All the hidden and output units started with an activity level of 0.2 at the beginning of each sequence. We used a version of the learning procedure in which the gradient of the error with respect to each weight is computed for a whole set of examples before the weights are changed. This means that each connection must accumulate the sum of the gradients for all the examples and for all the time steps involved in each example. During training, we used a particular set of 20 examples, and after these were learned almost perfectly we tested the network on the remaining examples to see if it had picked up on the obvious regularity that relates the first two items of a sequence to the subsequent four. The results are shown in Table 11. For four out of the five test sequences, the output units all have the correct values at all times (assuming we treat values above 0.5 as 1 and values below 0.5 as 0). The network has clearly captured the rule that the first item of a sequence determines the third and fourth, and the second determines the fifth and sixth. We repeated the simulation with a different set of random initial weights, and it got all five test sequences correct. The learning required 260 sweeps through all 20 training sequences. The errors in the output units were computed as follows: For a unit that should be on, there was no error if its activity level was above 0.8, otherwise the derivative of the error was the amount below 0.8. Similarly, for output units that should be off, the derivative of the error was the amount above 0.2. After each sweep, each weight was decremented by 0.02 times the total gradient accumulated on that sweep plus 0.9
乘以之前的权重变化。我们已经证明，该学习过程可用于创建一个
times the previous weight change. We have shown that the learning procedure can be used to create a
具有有趣序列行为的网络，但我们使用的特定问题可以通过简单地使用隐藏单元创建“延迟线”来解决，这些延迟线在固定长度时间内保持信息，然后允许其影响输出。一个更困难的问题，无法用固定持续时间的延迟线解决，如表 12 所示。输出与之前相同，但两个输入项可以在可变时间到达，因此例如在时间 2 到达的项可能是第一项或第二项，从而决定输出单元在第五和第六或第七和第八时间步的状态。新任务相当于要求一个缓冲区，它在可变时间接收两个输入“词”，并一个接一个地输出它们的“音位实现”。这个问题被一个与上述类似但具有 60 个隐藏单元并且随机省略了一半可能连接的网络成功解决。第 360 节基本机制
network with interesting sequential behavior, but the particular problem we used can be solved by simply using the hidden units to create "delay lines" which hold information for a fixed length of time before allowing it to influence the output. A harder problem that cannot be solved with delay lines of fixed duration is shown in Table 12. The output is the same as before, but the two input items can arrive at variable times so that the item arriving at time 2, for example, could be either the first or the second item and could therefore determine the states of the output units at either the fifth and sixth or the seventh and eighth times. The new task is equivalent to requiring a buffer that receives two input "words" at variable times and outputs their "phonemic realizations" one after the other. This problem was solved successfully by a network similar to the one above except that it had 60 hidden units and half of their possible connections omitted at random. The 360 BASIC MECHANISMS
网络在五个新测试序列上的性能 输入序列 A 期望输出 2 2
PERFORMANCE OF THE NETWORK ON FIVE NOVEL TEST SEQUENCES Input Sequence A Desired Outputs 2 2
Copyrighted Material 8. LEARNING INTERNAL R EPRESENTATIONS 3 5 9 and every output u n i t was also connected to every other output unit and to i tself. All the connecti ons started with small random weights uniformly distri buted between - 0. 3 and + 0. 3 . All the hidden and out put units started with an acti vi ty level of 0.2 at the beginning of each sequence. We used a version of the learni ng procedure i n wh ich the gradient of the error with respect to each weight is computed for a whole set of examples before the weights are changed . Th is means that each con nection must accum ulate the sum of the gradients for all the examples and for all the time steps i n vol ved i n each exam ple. During trai ni ng, we used a particular set of 20 examples , and after these were learned al most perfectly we tested the network on the remaining examples to see if it had picked up on the obvious regularity that relates the fi rst two i tems of a sequence to the subseq uent four. The results are shown in Table 1 1 . For fou r out of the fi ve test sequences , the output u n i ts all have the correct val ues at all ti mes (assu ming we treat val ues above 0.5 as 1 and values below 0.5 as 0) . The network has clearly captu red the rule that the fi rst item of a sequence determ i nes the third and fou rth , and the second determi nes the fi fth and si xth . We repeated the simulation with a differen t set of random i n i tial weights, and it got all fi ve test sequences correct . The learning requi red 260 sweeps through all 20 t rai ning seq uences. The errors in the ou tput u n i ts were compu ted as follows : For a unit that should be o n , there was no error if its acti vity level was above 0 . 8 , otherwise the deri vati ve o f the error was t h e amount below 0 . 8 . S i m i larly, for output uni ts that should b e off, the deri vat i ve of t h e error was the amount above 0. 2 . After each sweep, each weight was decremented by .02 t i mes the total gradient accumulated on that sweep plus 0 . 9
输出单元 2 0.2 0.16 0.13 0.82 0.88 0.03
Output Unit 2 0.2 0.16 0.13 0.82 0.88 0.03
输出单元 3 0.2 0.07 0.08 0.03 0.01 0.22
Output Unit 3 0.2 0.07 0.08 0.03 0.01 0.22
输出单元 1 0.2 0.12 0.20 0.25 0.48 0.26
Output Unit 1 0.2 0.12 0.20 0.25 0.48 0.26
输出单元 2 0.2 0.16 0.80 0.05 0.04 0.09
Output Unit 2 0.2 0.16 0.80 0.05 0.04 0.09
输出单元 3 0.2 0.07 0.02 0.79 0.48 0.53
Output Unit 3 0.2 0.07 0.02 0.79 0.48 0.53
输入序列 C A 期望输出 3 2
Input Sequence C A Desired Outputs 3 2
输出单元 1 0.2 0.12 0.19 0.80 0.87 0.11
Output Unit 1 0.2 0.12 0.19 0.80 0.87 0.11
输出单元 2 0.2 0.16 0.19 0.00 0.13 0.70
Output Unit 2 0.2 0.16 0.19 0.00 0.13 0.70
输出单元 3 0.2 0.07 0.80 0.13 0.01 0.25
Output Unit 3 0.2 0.07 0.80 0.13 0.01 0.25
输出单元 1 0.2 0.12 0.16 0.79 0.07 0.11
Output Unit 1 0.2 0.12 0.16 0.79 0.07 0.11
输出单元 2 0.2 0.16 0.80 0.15 0.87 0.05
Output Unit 2 0.2 0.16 0.80 0.15 0.87 0.05
输出单元 3 0.2 0.07 0.20 0.01 0.13 0.96
Output Unit 3 0.2 0.07 0.20 0.01 0.13 0.96
输出单元 1 0.2 0.12 0.80 0.09 0.27 0.78
Output Unit 1 0.2 0.12 0.80 0.09 0.27 0.78
输出单元 2 0.2 0.16 0.20 0.13 0.01 0.02
Output Unit 2 0.2 0.16 0.20 0.13 0.01 0.02
输出单元 3 0.2 0.07 0.07 0.94 0.76 0.13
Output Unit 3 0.2 0.07 0.07 0.94 0.76 0.13
学习速度要慢得多，需要对全部 136 个训练样本进行数千次遍历。在 14 个测试样本上也有少量错误，但泛化能力仍然很好，大多数测试序列被正确分类。（原文混乱部分：“co ll!RJ p)9tg Rfd'lieM »te rial 8.”）内部表示学习 361
Learning was much slower, requiring thousands of sweeps through all 136 training examples. There were also a few more errors on the 14 test examples, but the generalization was still good with most of the test sequences being correctly classified. (garbled original: "co ll!RJ p)9tg Rfd'lieM »te rial 8.") L E A R N I N G I N T E R N A L R E P R E S E N T A T I O N S 3 6 1
序列 EA 312 的变化：在可变时间呈现前两项所产生的序列 EA——1312
V A R I A T I O N S O F T H E S E Q U E N C E E A \ 3 1 2 P R O D U C E D B Y P R E S E N T I N G T H E F I R S T T W O I T E M S A T V A R I A B L E T I M E S E A - - 1 3 1 2
在他们对感知机的悲观讨论中，Minsky 和 Papert（1969）在书末终于讨论了多层机器。他们写道：感知机尽管（甚至因为！）其严重的局限性，已证明值得研究。它具有许多引人注目的特征：线性、有趣的学习定理、作为并行计算的清晰范式简单性。没有理由认为这些优点中的任何一个能延续到多层版本。尽管如此，我们认为阐明（或拒绝）我们关于扩展是无效的这种直觉判断是一个重要的研究问题。也许会发现某个强大的收敛定理，或者找到某个深刻的原因解释为什么多层机器未能产生有趣的“学习定理”。（第 231-232 页）尽管我们的学习结果并不能保证我们能对所有可解问题找到解决方案，但我们的分析和结果表明，实际上，误差传播方案在几乎所有情况下都能找到解决方案。简言之，我们相信我们已经回应了 Minsky 和 Papert 的挑战，并找到了一个足够强大的学习结果，证明他们对多层机器学习的悲观是站不住脚的。理解我们描述过程的一种方式是将其视为一台并行计算机，在向其展示了指定某个函数的适当输入/输出示例后，它自我编程以计算该函数。众所周知，并行计算机难以编程。这里我们拥有一种机制，无需实际知道如何编写程序就能让系统自行完成。Parker（1985）强调该材料属于 362 基本机制。
In their pessimistic discussion of perceptrons, Minsky and Papert (1969) finally discuss multilayer machines near the end of their book. They state: The perceptron has shown itself worthy of study despite (and even because of!) its severe limitations. It has many features that attract attention: its linearity; its intriguing learning theorem; its clear paradigmatic simplicity as a kind of parallel computation. There is no reason to suppose that any of these virtues carry over to the many-layered version. Nevertheless, we consider it to be an important research problem to elucidate (or reject) our intuitive judgement that the extension is sterile. Perhaps some powerful convergence theorem will be discovered, or some profound reason for the failure to produce an interesting "learning theorem" for the multilayered machine will be found. (pp. 231-232) Although our learning results do not guarantee that we can find a solution for all solvable problems, our analyses and results have shown that as a practical matter, the error propagation scheme leads to solutions in virtually every case. In short, we believe that we have answered Minsky and Papert's challenge and have found a learning result sufficiently powerful to demonstrate that their pessimism about learning in multilayer machines was misplaced. One way to view the procedure we have been describing is as a parallel computer that, having been shown the appropriate input/output exemplars specifying some function, programs itself to compute that function in general. Parallel computers are notoriously difficult to program. Here we have a mechanism whereby we do not actually have to know how to write the program in order to get the system to do it. Parker (1985) has emphasized that the material is 362 BASIC MECHANISMS.
许多时候，我们惊讶地通过观察学习算法的行为发现了计算有趣函数的新方法。这也引发了泛化问题。在上述大多数案例中，我们向系统展示了全部示例。有趣的问题是，如果我们只在训练时呈现示例的子集，然后观察系统对其余示例的泛化情况，会发生什么。在本文展示的小型问题中，系统有时会找到解。
On many occasions we have been surprised to learn of new methods of computing interesting functions by observing the behavior of our learning algorithm. This also raised the question of generalization. In most of the cases presented above, we have presented the system with the entire set of exemplars. It is interesting to ask what would happen if we presented only a subset of the exemplars at training time and then watched the system generalize to remaining exemplars. In small problems such as those presented here, the system sometimes finds solu
learn i ng was much sl ower, requiring thousands o f sweeps through all 1 3 6 trai ning exam ples . There were also a few more errors on the 1 4 test examples , but the general i zation was still good with most of the test sequences bei ng co ll!RJ p)9tg Rfd'lieM »te rial 8. LEA R N I NG INTE R N A L R E P R ES ENTATIONS 3 6 1
这些问题的解决方案无法正确泛化。然而，针对更大问题的初步结果在这方面非常令人鼓舞。这项研究仍在进行中，无法在此报告。这是我们目前一个非常活跃的研究兴趣。最后，我们应该说这项工作尚未完成。我们才刚刚开始研究循环网络和 sigma-pi 单元。我们尚未将学习程序应用于许多非常复杂的问题。然而，迄今为止的结果令人鼓舞，我们将继续工作。
Solutions to the problems which do not properly generalize. However, preliminary results on larger problems are very encouraging in this regard. This research is still in progress and cannot be reported here. This is currently a very active interest of ours. Finally, we should say that this work is not yet in a finished form. We have only begun our study of recurrent networks and sigma-pi units. We have not yet applied our learning procedure to many very complex problems. However, the results to date are encouraging and we are continuing our work.