A Fast Learning Algorithm for Deep Belief Nets
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→多伦多大学计算机科学系，国王学院路 10 号，多伦多，加拿大 M5S 3G4
Department of Computer Science University of Toronto 10 Kings College Road Toronto, Canada M5S 3G4
多伦多大学计算机科学系，国王学院路 10 号，多伦多，加拿大 M5S 3G4
Department of Computer Science University of Toronto 10 Kings College Road Toronto, Canada M5S 3G4
新加坡国立大学计算机科学系，新加坡科学大道 3 号，邮编 117543，电子邮箱：tehyw@comp.nus.edu.sg
Department of Computer Science National University of Singapore 3 Science Drive 3, Singapore, 117543 tehyw@comp.nus.edu.sg
我们展示了如何使用“互补先验”来消除解释效应，这些效应在具有许多隐藏层的密集连接信念网中使推理变得困难。利用互补先验，我们推导出一种快速的贪婪算法，该算法可以逐层学习深度、有向的信念网络，前提是顶层两层形成一个无向的联想记忆。快速的贪婪算法用于初始化一个较慢的学习过程，该过程使用对比版本的 wake-sleep 算法对权重进行微调。微调后，一个具有三个隐藏层的网络能够很好地生成手写数字图像及其标签的联合分布的生成模型。这个生成模型在手写数字分类上优于最好的判别式学习算法。数字所在的低维流形由顶层联想记忆自由能景观中的长沟壑建模，通过使用有向连接来展示联想记忆的所想，可以轻松探索这些沟壑。
We show how to use “complementary priors” to eliminate the explaining away effects that make inference difficult in densely-connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modelled by long ravines in the free-energy landscape of the top-level associative memory and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind.
在具有许多隐藏层的密集连接有向信念网络中，学习是困难的，因为当给定一个数据向量时，很难推断隐藏活动的条件分布。变分方法使用简单近似来逼近真实条件分布，但近似可能较差，尤其是在最深的隐藏层，其中先验假设独立性。此外，变分学习仍然需要所有参数一起学习，并且随着参数数量的增加，学习时间规模变得很差。我们描述了一个模型，其中顶部两个隐藏层形成一个无向联想记忆（见图 1），而
Learning is difficult in densely-connected, directed belief nets that have many hidden layers because it is difficult to infer the conditional distribution of the hidden activities when given a data vector. Variational methods use simple approximations to the true conditional distribution, but the approximations may be poor, especially at the deepest hidden layer where the prior assumes independence. Also, variational learning still requires all of the parameters to be learned together and makes the learning time scale poorly as the number of parameters increases. We describe a model in which the top two hidden layers form an undirected associative memory (see figure 1) and the
剩余的隐藏层形成一个有向无环图，将联想记忆中的表示转换为可观测变量，如图像的像素。这种混合模型具有一些吸引人的特点：1. 存在一种快速、贪婪的学习算法，即使在具有数百万参数和许多隐藏层的深度网络中，也能快速找到相当好的参数集。2. 学习算法是无监督的，但可以通过学习一个同时生成标签和数据的模型应用于标记数据。3. 存在一种微调算法，学习一个优秀的生成模型，在手写数字 MNIST 数据库上优于判别方法。4. 生成模型使得深度隐藏层中的分布式表示易于解释。5. 形成感知所需的推理既快速又准确。6. 学习算法是局部的：突触强度的调整仅取决于突触前和突触后神经元的状态。7. 通信简单：神经元只需传递其随机二值状态。第 2 节介绍了“互补”先验的思想，它精确抵消了使有向模型中推理困难的“解释消除”现象。给出了一个具有互补先验的有向信念网络的例子。第 3 节展示了受限玻尔兹曼机与具有权值共享的无限有向网络之间的等价性。第 4 节介绍了一种快速、贪婪的学习算法，用于一次一层地构建多层有向网络。利用变分界，它表明随着每一新层的添加，整体生成模型得到改进。该贪婪算法在重复使用相同的“弱”学习器方面与提升有些相似，但不是重新加权每个数据向量以确保下一步学习新内容，而是重新表示它。具有 2000 个顶层单元的“弱”学习器
remaining hidden layers form a directed acyclic graph that converts the representations in the associative memory into observable variables such as the pixels of an image. This hybrid model has some attractive features: 1. There is a fast, greedy learning algorithm that can find a fairly good set of parameters quickly, even in deep networks with millions of parameters and many hidden layers. 2. The learning algorithm is unsupervised but can be applied to labeled data by learning a model that generates both the label and the data. 3. There is a fine-tuning algorithm that learns an excellent generative model which outperforms discriminative methods on the MNIST database of hand-written digits. 4. The generative model makes it easy to interpret the distributed representations in the deep hidden layers. 5. The inference required for forming a percept is both fast and accurate. 6. The learning algorithm is local: adjustments to a synapse strength depend only on the states of the presynaptic and postsynaptic neuron. 7. The communication is simple: neurons only need to communicate their stochastic binary states. Section 2 introduces the idea of a “complementary” prior which exactly cancels the “explaining away” phenomenon that makes inference difficult in directed models. An example of a directed belief network with complementary priors is presented. Section 3 shows the equivalence between restricted Boltzmann machines and infinite directed networks with tied weights. Section 4 introduces a fast, greedy learning algorithm for constructing multi-layer directed networks one layer at a time. Using a variational bound it shows that as each new layer is added, the overall generative model improves. The greedy algorithm bears some resemblance to boosting in its repeated use of the same “weak” learner, but instead of re-weighting each data-vector to ensure that the next step learns something new, it re-represents it. The “weak” learner that 2000 top-level units
图 1：用于建模数字图像和数字标签联合分布的网络。在本文中，每个训练样本由一张图像和一个明确的类别标签组成，但进行中的工作表明，如果将“标签”替换为一个多层通路，其输入是多个不同说话者说孤立数字的频谱图，则可以使用相同的学习算法。然后，网络学习生成由同一数字类别的图像和频谱图组成的配对。用于构建深度有向网络的网络本身是一个无向图模型。第 5 节展示了如何使用“上下”算法对快速贪心算法产生的权重进行微调。这是 Hinton 等人(1995)的唤醒-睡眠算法的对比版本，它不会遭受“模式平均”问题，该问题可能导致唤醒-睡眠算法学习到较差的识别权重。第 6 节展示了一个具有三个隐藏层和约 170 万个权重的网络在 MNIST 手写数字集上的模式识别性能。当未提供几何知识且没有特殊预处理时，该网络在 10000 个数字的官方测试集上的泛化性能为 1.25%的错误率。这优于针对特定应用未进行手工调整的最佳反向传播网络所达到的 1.5%。它也略优于 Decoste 和 Schoelkopf (2002)在相同任务上支持向量机报告的 1.4%错误率。最后，第 7 节展示了当网络不受视觉输入约束时，其“心智”中发生的情况。该网络具有完整的生成模型，因此很容易窥视其“心智”——我们只需从其高层表示生成一张图像。在整篇论文中，我们将考虑由……组成的网络。
Figure 1: The network used to model the joint distribution of digit images and digit labels. In this paper, each training case consists of an image and an explicit class label, but work in progress has shown that the same learning algorithm can be used if the “labels” are replaced by a multilayer pathway whose inputs are spectrograms from multiple different speakers saying isolated digits. The network then learns to generate pairs that consist of an image and a spectrogram of the same digit class. The network used to construct deep directed nets is itself an undirected graphical model. Section 5 shows how the weights produced by the fast greedy algorithm can be fine-tuned using the “up-down” algorithm. This is a contrastive version of the wake-sleep algorithm Hinton et al. (1995) that does not suffer from the “mode-averaging” problems that can cause the wake-sleep algorithm to learn poor recognition weights. Section 6 shows the pattern recognition performance of a network with three hidden layers and about 1.7 million weights on the MNIST set of handwritten digits. When no knowledge of geometry is provided and there is no special preprocessing, the generalization performance of the network is 1.25% errors on the 10,000 digit official test set. This beats the 1.5% achieved by the best back-propagation nets when they are not hand-crafted for this particular application. It is also slightly better than the 1.4% errors reported by Decoste and Schoelkopf (2002) for support vector machines on the same task. Finally, section 7 shows what happens in the mind of the network when it is running without being constrained by visual input. The network has a full generative model, so it is easy to look into its mind – we simply generate an image from its high-level representations. Throughout the paper, we will consider nets composed of
图 2：一个简单的逻辑信念网络，包含两个独立、罕见的原因，当我们观察到房子跳跃时，它们变得高度反相关。地震节点上的偏置为-10 意味着，在没有任何观测的情况下，该节点关闭的可能性比开启的可能性大\(e^{10}\)倍。如果地震节点开启而卡车节点关闭，则跳跃节点的总输入为 0，这意味着它开启的概率为一半。对于房子跳跃的观测，这比两个隐藏原因均未激活时的几率\(e^{-20}\)是一个更好的解释。但为了解释观测而同时开启两个隐藏原因是浪费的，因为两者同时发生的概率为\(e^{-10} \times e^{-10} = e^{-20}\)。当地震节点开启时，它“解释消除”了卡车节点的证据。这些是随机二元变量，但该思想可以推广到其他模型，其中变量的对数概率是其直接连接邻居状态的加性函数（详见附录 A）。
Figure 2: A simple logistic belief net containing two independent, rare causes that become highly anti-correlated when we observe the house jumping. The bias of -10 on the earthquake node means that, in the absence of any observation, this node is \(e^{10}\) times more likely to be off than on. If the earthquake node is on and the truck node is off, the jump node has a total input of 0 which means that it has an even chance of being on. This is a much better explanation of the observation that the house jumped than the odds of \(e^{-20}\) which apply if neither of the hidden causes is active. But it is wasteful to turn on both hidden causes to explain the observation because the probability of them both happening is \(e^{-10} \times e^{-10} = e^{-20}\). When the earthquake node is turned on it “explains away” the evidence for the truck node. These are stochastic binary variables but the ideas can be generalized to other models in which the log probability of a variable is an additive function of the states of its directly-connected neighbours (see Appendix A for details).
解释消除现象（如图 2 所示）使得有向信念网络中的推理变得困难。在密集连接的网络中，除混合模型或具有加性高斯噪声的线性模型等少数特殊情况外，隐藏变量的后验分布是难以处理的。马尔可夫链蒙特卡洛方法（Neal, 1992）可用于从后验中采样，但通常非常耗时。变分方法（Neal and Hinton, 1998）用更易处理的分布近似真实后验，并可用于改进训练数据对数概率的下界。令人欣慰的是，即使隐藏状态的推理不正确，学习也能保证改进变分下界，但更好的方法是找到一种完全消除解释消除的方法，即使在隐藏变量对可见变量有高度相关影响的模型中也是如此。人们普遍认为这是不可能的。逻辑斯蒂信念网络（Neal, 1992）由随机二元单元组成。当网络用于生成数据时，单元 i 被激活的概率是其直接祖先 j 的状态以及来自祖先的有向连接上的权重 w_{ij}的逻辑斯蒂函数：\(p(s_i=1) = \frac{1}{1+\exp(-b_i - \sum_j s_j w_{ij})}\ (1) 其中 b_i 是单元 i 的偏置。如果逻辑斯蒂信念网络只有一个隐藏层，那么隐藏变量上的先验分布是阶乘的，因为当模型用于生成数据时，它们的二元状态是独立选择的。后验分布中的非独立性是由来自数据的似然项造成的。也许我们可以通过使用额外的隐藏层来创建一个与似然项中的相关性完全相反的“互补”先验，从而消除第一个隐藏层中的解释消除。这样，当似然项与先验相乘时，我们将得到一个完全阶乘的后验。互补先验是否存在并不明显，但图 3 显示了一个具有共享权重的无限逻辑斯蒂信念网络的简单示例，其中先验在每个隐藏层都是互补的（关于互补先验存在条件的更一般处理，请参见附录 A）。使用共享权重来构建互补先验似乎只是一种使有向模型等价于无向模型的技巧。然而，正如我们将看到的，它产生了一种新颖且非常高效的学习算法，该算法通过逐步解耦每个层的权重与更高层的权重来工作。
The phenomenon of explaining away (illustrated in figure 2) makes inference difficult in directed belief nets. In densely connected networks, the posterior distribution over the hidden variables is intractable except in a few special cases such as mixture models or linear models with additive Gaussian noise. Markov Chain Monte Carlo methods (Neal, 1992) can be used to sample from the posterior, but they are typically very time consuming. Variational methods (Neal and Hinton, 1998) approximate the true posterior with a more tractable distribution and they can be used to improve a lower bound on the log probability of the training data. It is comforting that learning is guaranteed to improve a variational bound even when the inference of the hidden states is done incorrectly, but it would be much better to find a way of eliminating explaining away altogether, even in models whose hidden variables have highly correlated effects on the visible variables. It is widely assumed that this is impossible. A logistic belief net (Neal, 1992) is composed of stochastic binary units. When the net is used to generate data, the probability of turning on unit i is a logistic function of the states of its immediate ancestors, j, and of the weights, w_{ij}, on the directed connections from the ancestors: \(p(s_i=1) = \frac{1}{1+\exp(-b_i - \sum_j s_j w_{ij})}\ (1) where b_i is the bias of unit i. If a logistic belief net only has one hidden layer, the prior distribution over the hidden variables is factorial because their binary states are chosen independently when the model is used to generate data. The non-independence in the posterior distribution is created by the likelihood term coming from the data. Perhaps we could eliminate explaining away in the first hidden layer by using extra hidden layers to create a “complementary” prior that has exactly the opposite correlations to those in the likelihood term. Then, when the likelihood term is multiplied by the prior, we will get a posterior that is exactly factorial. It is not at all obvious that complementary priors exist, but figure 3 shows a simple example of an infinite logistic belief net with tied weights in which the priors are complementary at every hidden layer (see Appendix A for a more general treatment of the conditions under which complementary priors exist). The use of tied weights to construct complementary priors may seem like a mere trick for making directed models equivalent to undirected ones. As we shall see, however, it leads to a novel and very efficient learning algorithm that works by progressively untying the weights in each layer from the weights in higher layers.
The phenomenon of explaining away (illustrated in figure 2) makes inference difficult in directed belief nets. In densely connected networks, the posterior distribution over the hid-den variables is intractable except in a few special cases such as mixture models or linear models with additive Gaussian noise. Markov Chain Monte Carlo methods (Neal, 1992) can be used to sample from the posterior, but they are typically very time consuming. Variational methods (Neal and Hinton, 1998) approximate the true posterior with a more tractable distribution and they can be used to improve a lower bound on the log probability of the training data. It is comforting that learning is guaranteed to improve a variational bound even when the inference of the hidden states is done incorrectly, but it would be much better to find a way of eliminating ex-plaining away altogether, even in models whose hidden vari-ables have highly correlated effects on the visible variables. It is widely assumed that this is impossible. A logistic belief net (Neal, 1992) is composed of stochas-tic binary units. When the net is used to generate data, the probability of turning on unit i is a logistic function of the states of its immediate ancestors, j, and of the weights, wij ,on the directed connections from the ancestors:
我们可以从图 3 所示的无限有向网络中生成数据，方法是从无限深的隐藏层 1 的随机配置开始，然后执行自顶向下的“祖先”传递，其中每个层中每个变量的二元状态由来自其上层活跃父节点的自顶向下输入所确定的伯努利分布决定。在这方面，它就像任何其他有向无环信念网络一样。然而，与其他有向网络不同，我们可以通过从可见单元上的数据向量开始，然后使用转置权重矩阵依次推断每个隐藏层上的因式分解分布，来对所有隐藏层的真实后验分布进行采样。在每个隐藏层，我们在计算上一层 2 的因式分解后验之前，先对因式分解后验进行采样。附录 A 表明，该过程产生无偏样本，因为每一层的互补先验确保后验分布确实是因式分解的。由于我们可以从真实后验中采样，我们可以计算数据对数概率的导数。令
We can generate data from the infinite directed net in figure 3 by starting with a random configuration at an infinitely deep hidden layer 1 and then performing a top-down “ancestral” pass in which the binary state of each variable in a layer is chosen from the Bernoulli distribution determined by the top-down input coming from its active parents in the layer above. In this respect, it is just like any other directed acyclic belief net. Unlike other directed nets, however, we can sample from the true posterior distribution over all of the hidden layers by starting with a data vector on the visible units and then using the transposed weight matrices to infer the factorial distributions over each hidden layer in turn. At each hidden layer we sample from the factorial posterior before computing the factorial posterior for the layer above 2. Appendix A shows that this procedure gives unbiased samples because the complementary prior at each layer ensures that the posterior distribution really is factorial. Since we can sample from the true posterior, we can compute the derivatives of the log probability of the data. Let
我们开始计算生成权重的导数，
us start by computing the derivative for a generative weight,
该权重从层 H0 中的单元 j 到层 V0 中的单元 i（见图 3）。在逻辑信念网络中，对于单个数据向量 v0 的最大似然学习规则为：
, from a unit j in layer H0 to unit i in layer V0 (see figure 3). In a logistic belief net, the maximum likelihood learning rule for a single data-vector, v0, is:
) (2) 其中\(⟨·⟩\)表示对采样状态的平均，且
) (2) where \(⟨·⟩\) denotes an average over the sampled states and
是如果从采样的隐藏状态随机重建可见向量时单元 i 被开启的概率。从第一隐藏层 H0 的采样二元状态计算第二隐藏层 V1 上的后验分布，与重建数据的过程完全相同，因此 v1
is the probability that unit i would be turned on if the visible vector was stochastically reconstructed from the sampled hidden states. Computing the posterior distribution over the second hidden layer, V1, from the sampled binary states in the first hidden layer, H0, is exactly the same process as reconstructing the data, so v1
是一个从伯努利随机变量中采样的样本，其概率为 \( \hat{v}_0 \)
is a sample from a Bernoulli random variable with probability \( \hat{v}_0 \)
。因此，学习规则可以写为：
. The learning rule can therefore be written as:
在从方程(2)推导方程(3)时是无问题的，因为 \( \hat{v}_0 \)
is unproblematic in the derivation of Eq. 3 from Eq. 2 because \( \hat{v}_0 \)
是一个条件于 \( h_0 \) 的期望
is an expectation that is conditional on \( h_0 \)
。由于权重是复制的，生成权重的完整导数是通过对所有层对之间的生成权重导数求和得到的：
. Since the weights are replicated, the full derivative for a generative weight is obtained by summing the derivatives of the generative weights between all pairs of layers:
+... (4) 所有垂直对齐的项相互抵消，留下玻尔兹曼机学习规则，如公式 (5) 所示。
+... (4) All of the vertically aligned terms cancel, leaving the Boltzmann machine learning rule of Eq. (5).
可能并不立即显而易见，图 3 中的无限有向网络等价于受限玻尔兹曼机（RBM）。RBM 具有单层隐藏单元，这些单元彼此不连接，并与一层可见单元具有无向的对称连接。为了从 RBM 生成数据，我们可以从其中一层的随机状态开始，然后执行交替吉布斯采样：一层中的所有单元根据另一层中单元的当前状态并行更新，并重复此过程，直到系统从平衡分布中采样。注意，这与使用绑定权重的无限信念网络生成数据的过程完全相同。为了在 RBM 中执行最大似然学习，我们可以使用两个相关性之间的差异。对于每个权重，
It may not be immediately obvious that the infinite directed net in figure 3 is equivalent to a Restricted Boltzmann Machine (RBM). An RBM has a single layer of hidden units which are not connected to each other and have undirected, symmetrical connections to a layer of visible units. To generate data from an RBM, we can start with a random state in one of the layers and then perform alternating Gibbs sampling: All of the units in one layer are updated in parallel given the current states of the units in the other layer and this is repeated until the system is sampling from its equilibrium distribution. Notice that this is exactly the same process as generating data from the infinite belief net with tied weights. To perform maximum likelihood learning in an RBM, we can use the difference between two correlations. For each weight,
wij，位于可见单元 i 和隐藏单元 j 之间，我们测量相关性<v_i^0 h_j^0>。
wij, between a visible unit i and a hidden unit j, we measure the correlation <v_i^0 h_j^0>.
图 3：一个具有共享权重的无限逻辑信念网络。向下的箭头代表生成模型。向上的箭头不属于模型本身，它们表示当数据向量被固定于 V0 时，用于从网络每一隐藏层的后验分布中推断样本的参数。可见单元和隐藏状态从其条件分布中采样，该分布是因子化的。然后，使用交替吉布斯采样，我们运行如图 4 所示的马尔可夫链直到其达到平稳分布，并测量数据分布 \(P_0\) 与模型定义的平衡分布 \(P_\infty\) 之间的相关性 \(v_\infty\)。
Figure 3: An infinite logistic belief net with tied weights. The downward arrows represent the generative model. The upward arrows are not part of the model. They represent the parameters that are used to infer samples from the posterior distribution at each hidden layer of the net when a data vector is clamped on V0. The visible units and the hidden states are sampled from their conditional distribution, which is factorial. Then, using alternating Gibbs sampling, we run the Markov chain shown in Figure 4 until it reaches its stationary distribution and measure the correlation \(v_\infty\), between the distribution of the data, \(P_0\), and the equilibrium distribution defined by the model, \(P_\infty\).
在对比散度学习（Hinton, 2002）中，我们仅在测量第二次相关性之前运行马尔可夫链 n 个完整步骤。这等价于忽略导数。
In contrastive divergence learning (Hinton, 2002), we only run the Markov chain for n full steps before measuring the second correlation. This is equivalent to ignoring the derivatives.
Figure 3: An infinite logistic belief net with tied weights. The downward arrows represent the generative model. The up-ward arrows are not part of the model. They represent the parameters that are used to infer samples from the posterior distribution at each hidden layer of the net when a datavector is clamped on V0.the visible units and the hidden states are sampled from their conditional distribution, which is factorial. Then, using al-ternating Gibbs sampling, we run the Markov chain shown in figure 4 until it reaches its stationary distribution and measure the correlation <v ∞
图 4 描述了一个使用交替吉布斯采样的马尔可夫链。在吉布斯采样的一个完整步骤中，顶层的隐藏单元通过将式(1)应用于从底层可见单元的当前状态接收的输入来并行更新，然后根据当前的隐藏状态并行更新所有可见单元。该链通过将可见单元的二进制状态设置为与数据向量相同来初始化。在隐藏单元第一次更新后以及链的末尾，测量可见单元和隐藏单元活动之间的相关性。这两个相关性的差值提供了更新连接权重的学习信号，这些连接来自无限网络的更高层。所有这些被忽略的导数的和是层\(V_n\)中后验分布的对数概率的导数，也是层\(V_n\)中的后验分布\(P_n^\theta\)与模型定义的平衡分布之间的 KL 散度的导数。因此对比散度学习最小化两个 KL 散度之差：
Figure 4: This depicts a Markov chain that uses alternating Gibbs sampling. In one full step of Gibbs sampling, the hidden units in the top layer are all updated in parallel by applying Eq. 1 to the inputs received from the current states of the visible units in the bottom layer, then the visible units are all updated in parallel given the current hidden states. The chain is initialized by setting the binary states of the visible units to be the same as a data-vector. The correlations in the activities of a visible and a hidden unit are measured after the first update of the hidden units and again at the end of the chain. The difference of these two correlations provides the learning signal for updating the weight on the connection that come from the higher layers of the infinite net. The sum of all these ignored derivatives is the derivative of the \log\ probability of the posterior distribution in layer \(V_n\), which is also the derivative of the Kullback-Leibler divergence between the posterior distribution in layer \(V_n\), \(P_n^\theta\), and the equilibrium distribution defined by the model. So contrastive divergence learning minimizes the difference of two Kullback-Leibler divergences:
)(6) 忽略采样噪声，这个差值永远不会为负，因为吉布斯采样用于从\(P_0\)产生\(P_n^\theta\)，而吉布斯采样总是减小与平衡分布的 KL 散度。重要的是注意到\(P_n^\theta\)依赖于当前模型参数，并且对比散度学习忽略了\(P_n^\theta\)随参数变化的方式。这个问题不会出现在\(P_0\)上，因为训练数据不依赖于参数。关于最大似然和对比散度学习规则之间关系的实证研究见于 Carreira-Perpinan 和 Hinton (2005)。受限玻尔兹曼机中的对比散度学习效率足够高，因此实用（Mayraz 和 Hinton, 2001）。使用实值单元和不同采样方案的变体在 Teh 等人(2003)中描述，并已成功用于地形图的形成建模（Welling 等人, 2003）、自然图像去噪（Roth 和 Black, 2005）或生物细胞图像去噪（Ning 等人, 2005）。Marks 和 Movellan (2001)描述了一种使用对比散度进行因子分析的方法，Welling 等人(2005)表明，具有逻辑、二进制可见单元和线性、高斯隐藏单元的网络可用于快速文档检索。然而，效率似乎付出了高昂的代价：当以明显的方式应用时，对比散度学习对于每一层具有不同权重的深度多层网络失败，因为这些网络甚至需要太长时间才能达到带钳制数据向量的条件平衡。我们现在展示，RBM 与具有共享权重的无限有向网络之间的等价性，为权重不共享的多层网络提供了一种高效的学习算法。
) (6) Ignoring sampling noise, this difference is never negative because Gibbs sampling is used to produce \(P_n^\theta\) from \(P_0\) and Gibbs sampling always reduces the Kullback-Leibler divergence with the equilibrium distribution. It is important to notice that \(P_n^\theta\) depends on the current model parameters and the way in which \(P_n^\theta\) changes as the parameters change is being ignored by contrastive divergence learning. This problem does not arise with \(P_0\) because the training data does not depend on the parameters. An empirical investigation of the relationship between the maximum likelihood and the contrastive divergence learning rules can be found in Carreira-Perpinan and Hinton (2005). Contrastive divergence learning in a restricted Boltzmann machine is efficient enough to be practical (Mayraz and Hinton, 2001). Variations that use real-valued units and different sampling schemes are described in Teh et al. (2003) and have been quite successful for modeling the formation of topographic maps (Welling et al., 2003), for denoising natural images (Roth and Black, 2005) or images of biological cells (Ning et al., 2005). Marks and Movellan (2001) describe a way of using contrastive divergence to perform factor analysis and Welling et al. (2005) show that a network with logistic, binary visible units and linear, Gaussian hidden units can be used for rapid document retrieval. However, it appears that the efficiency has been bought at a high price: When applied in the obvious way, contrastive divergence learning fails for deep, multilayer networks with different weights at each layer because these networks take far too long even to reach conditional equilibrium with a clamped data-vector. We now show that the equivalence between RBM's and infinite directed nets with tied weights suggests an efficient learning algorithm for multilayer networks in which the weights are not tied.
一种高效学习复杂模型的方法是依次组合一组更简单的模型。为了强制序列中的每个模型学到与先前模型不同的内容，在每个模型学习完成后，以某种方式修改数据。在 Boosting（Freund, 1995）中，序列中的每个模型都在重新加权后的数据上训练，这些数据强调先前模型出错的案例。在主成分分析的一个版本中，建模方向上的方差被移除，从而迫使下一个建模方向位于正交子空间中（Sanger, 1989）。在投影追踪（Friedman and Stuetzle, 1981）中，通过非线性扭曲数据空间中的一个方向来变换数据，以去除该方向上的所有非高斯性。我们贪婪算法背后的思想是允许序列中的每个模型接收数据的不同表示。该模型对其输入向量执行非线性变换，并输出将用作序列中下一个模型输入的向量。图 5 展示了一个多层生成模型，其中顶部两层通过无向连接相互作用，而所有其他连接都是有向的。顶部的无向连接等价于具有无限多权重绑定的更高层。没有层内连接，并且为了简化分析，所有层具有相同数量的单元。通过假设高层之间的参数将用于构建 \(W_0\) 的互补先验，可以学习到 \(W_0\) 的合理（尽管不是最优）值。这等价于假设所有权重矩阵都被约束为相等。在此假设下学习 \(W_0\) 的任务简化为学习 RBM 的任务，尽管这仍然困难，但通过最小化对比散度可以快速找到良好的近似解。一旦 \(W_0\) 被学习，数据可以通过 \(W^T\) 映射
An efficient way to learn a complicated model is to combine a set of simpler models that are learned sequentially. To force each model in the sequence to learn something different from the previous models, the data is modified in some way after each model has been learned. In boosting (Freund, 1995), each model in the sequence is trained on re-weighted data that emphasizes the cases that the preceding models got wrong. In one version of principal components analysis, the variance in a modeled direction is removed thus forcing the next modeled direction to lie in the orthogonal subspace (Sanger, 1989). In projection pursuit (Friedman and Stuetzle, 1981), the data is transformed by nonlinearly distorting one direction in the data-space to remove all non-Gaussianity in that direction. The idea behind our greedy algorithm is to allow each model in the sequence to receive a different representation of the data. The model performs a non-linear transformation on its input vectors and produces as output the vectors that will be used as input for the next model in the sequence. Figure 5 shows a multilayer generative model in which the top two layers interact via undirected connections and all of the other connections are directed. The undirected connections at the top are equivalent to having infinitely many higher layers with tied weights. There are no intra-layer connections and, to simplify the analysis, all layers have the same number of units. It is possible to learn sensible (though not optimal) values for the parameters \(W_0\) by assuming that the parameters between higher layers will be used to construct a complementary prior for \(W_0\). This is equivalent to assuming that all of the weight matrices are constrained to be equal. The task of learning \(W_0\) under this assumption reduces to the task of learning an RBM and although this is still difficult, good approximate solutions can be found rapidly by minimizing contrastive divergence. Once \(W_0\) has been learned, the data can be mapped through \(W^T\)
到第一个隐藏层，以创建更高层次的“数据”。如果 RBM 是原始数据的完美模型，那么高层权重矩阵将已经完美地对高层“数据”进行了建模。然而，通常 RBM 无法完美地对原始数据进行建模，我们可以使用以下贪婪算法来改进生成模型： 1. 假设所有权重矩阵绑定，学习 \(W_0\)。 2. 冻结 \(W_0\)，并承诺使用 \(W^T\)
to create higher-level “data” at the first hidden layer. If the RBM is a perfect model of the original data, the higher-level “data” will already be modeled perfectly by the higher-level weight matrices. Generally, however, the RBM will not be able to model the original data perfectly and we can make the generative model better using the following greedy algorithm: 1. Learn \(W_0\) assuming all the weight matrices are tied. 2. Freeze \(W_0\) and commit ourselves to using \(W^T\)
图 5：一个混合网络。顶部两层具有无向连接，形成联想记忆。下面的层具有有向的、自上而下的生成连接，可用于将联想记忆的状态映射到图像。还有有向的、自下而上的识别连接，用于从下一层的二元活动推断某一层的因子化表示。在贪婪的初始学习中，识别连接与生成连接绑定。即使后续高层权重的变化意味着该推断方法不再正确，也使用因子化的近似后验分布来表示第一个隐藏层中变量的状态。 3. 保持所有高层权重矩阵彼此绑定，但与 \(W_0\) 解绑，学习一个 RBM 模型，用于通过使用 \(W^T\) 产生的高层“数据”
Figure 5: A hybrid network. The top two layers have undirected connections and form an associative memory. The layers below have directed, top-down, generative connections that can be used to map a state of the associative memory to an image. There are also directed, bottom-up, recognition connections that are used to infer a factorial representation in one layer from the binary activities in the layer below. In the greedy initial learning the recognition connections are tied to the generative connections. factorial approximate posterior distributions over the states of the variables in the first hidden layer, even if subsequent changes in higher level weights mean that this inference method is no longer correct. 3. Keeping all the higher weight matrices tied to each other, but untied from \(W_0\), learn an RBM model of the higher-level “data” that was produced by using \(W^T\)
来变换原始数据。如果该贪婪算法改变了高层权重矩阵，则保证能改进生成模型。如（Neal and Hinton, 1998）所示，在多层生成模型下，单个数据向量 \(v_0\) 的负对数概率受变分自由能的约束，该自由能是近似分布下的期望能量减去该分布的熵。
to transform the original data. If this greedy algorithm changes the higher-level weight matrices, it is guaranteed to improve the generative model. As shown in (Neal and Hinton, 1998), the negative log probability of a single data-vector, \(v_0\), under the multilayer generative model is bounded by a variational free energy which is the expected energy under the approximating distribution,
即 \(Q(h_0|v_0)\) 减去该分布的熵。对于有向模型，构型 \(v_0, h_0\) 的“能量”由下式给出：
\(Q(h_0|v_0)\), minus the entropy of that distribution. For a directed model, the “energy” of the configuration \(v_0, h_0\) is given by:
E(v0, h0) = - [log p(h0) + log p(v0|h0)] (7) 因此下界为：
E(v0, h0) = - [\log p(h0) + \log p(v0|h0)] (7) So the bound is:
Q(h0|v0) log Q(h0|v0) (8) 其中 h0 是第一隐藏层单元的二进制配置，p(h0) 是当前模型下 h0 的先验概率（由 H0 之上的权重定义），
Q(h0|v0) \log Q(h0|v0) (8) where h0 is a binary configuration of the units in the first hidden layer, p(h0) is the prior probability of h0 under the current model (which is defined by the weights above H0) and
Q(·| v0) 是第一隐藏层二进制配置上的任意概率分布。当且仅当 Q(·| v0) 是真实后验分布时，该下界成为等式。当所有权重矩阵绑定时，通过应用 W^T 在 H0 上产生的因子分布
Q(·| v0) is any probability distribution over the binary configurations in the first hidden layer. The bound becomes an equality if and only if Q(·| v0) is the true posterior distribution. When all of the weight matrices are tied together, the factorial distribution over H0 produced by applying \(W^T\)
作用于数据向量得到真实后验分布，因此在贪婪算法的第二步中，log p(v0) 等于该下界。第二步固定了 Q(·| v0) 和 p(v0|h0)，在这些项固定的情况下，下界的导数与 ∑ 的导数相同
to a data-vector is the true posterior distribution, so at step 2 of the greedy algorithm \log p(v0) is equal to the bound. Step 2 freezes both Q(·| v0) and p(v0|h0) and with these terms fixed, the derivative of the bound is the same as the derivative of \∑
Q(h0|v0) log p(h0) (9) 因此相对于更高层权重最大化下界，等价于最大化一个数据集的 log 概率，其中 h0 以概率 Q(h0|v0) 出现。如果下界变得更紧，即使 log p(v0) 的下界增加，log p(v0) 仍有可能下降，但
Q(h0|v0) \log p(h0) (9) So maximizing the bound w.r.t. the weights in the higher layers is exactly equivalent to maximizing the log probability of a dataset in which h0 occurs with probability Q(h0|v0). If the bound becomes tighter, it is possible for \log p(v0) to fall even though the lower bound on it increases, but \log p(v0)
它永远不会低于贪婪算法第 2 步时的值，因为此时边界是紧的，并且边界总是增加的。贪婪算法显然可以递归应用，因此如果我们使用完整最大似然玻尔兹曼机学习算法来学习每一组共享权值，然后将该组的最底层与上层的权值解绑，我们就可以逐层学习权值，并保证永远不会降低完整生成模型下数据的对数概率。在实践中，我们用对比散度学习替代最大似然玻尔兹曼机学习算法，因为它效果好且速度快得多。使用对比散度会使得该保证失效，但知道若以足够耐心学习每一层，额外层保证能改进不完美模型，这仍然令人安心。为了保证通过贪婪学习更多层来改进生成模型，考虑所有层大小相同的模型会很方便，这样高层权值就可以初始化为从下层解绑前学习到的值。然而，同样的贪婪算法甚至可以应用于层大小不同的情况。
can never fall below its value at step 2 of the greedy algorithm because the bound is tight at this point and the bound always increases. The greedy algorithm can clearly be applied recursively, so if we use the full maximum likelihood Boltzmann machine learning algorithm to learn each set of tied weights and then we untie the bottom layer of the set from the weights above, we can learn the weights one layer at a time with a guarantee that we will never decrease the log probability of the data under the full generative model. In practice, we replace the maximum likelihood Boltzmann machine learning algorithm with contrastive divergence learning because it works well and is much faster. The use of contrastive divergence voids the guarantee, but it is still reassuring to know that extra layers are guaranteed to improve imperfect models if we learn each layer with sufficient patience. To guarantee that the generative model is improved by greedily learning more layers, it is convenient to consider models in which all layers are the same size so that the higher-level weights can be initialized to the values learned before they are untied from the weights in the layer below. The same greedy algorithm, however, can be applied even when the layers are different sizes.
逐层学习权重矩阵是高效的，但并非最优。一旦高层权重被学习，权重和简单推理过程对于低层都不是最优的。贪婪学习产生的次优性对于诸如 Boosting 等监督方法相对无害。标签通常稀缺，且每个
Learning the weight matrices one layer at a time is efficient but not optimal. Once the weights in higher layers have been learned, neither the weights nor the simple inference procedure are optimal for the lower layers. The sub-optimality produced by greedy learning is relatively innocuous for supervised methods like boosting. Labels are often scarce and each
标签可能仅提供几个比特的参数约束，因此过拟合通常比欠拟合更成问题。回过头来重新拟合早期模型可能弊大于利。然而，无监督方法可以使用非常大的无标签数据集，且每个样本可能具有很高的维度，从而为生成模型提供许多比特的约束。此时欠拟合是一个严重问题，可以通过后续的反向拟合阶段来缓解，该阶段中先学习的权重被修正，以更好地与后学习的权重相配合。在贪婪地学习了每一层权重的良好初始值之后，我们将用于推理的“识别”权重与定义模型的“生成”权重解绑，但保留每一层后验必须用因子分布来近似的限制，其中层内变量在给定下层变量值的条件下条件独立。随后可以使用 Hinton 等人（1995）描述的 wake-sleep 算法的一个变体，让高层权重影响低层权重。在“上行阶段”（up-pass）中，识别权重用于从下到上的传递，随机地为每个隐藏变量选择一个状态。然后使用式（2.5）中的最大似然学习规则调整有向连接上的生成权重。顶层无向连接的权重如前所述，通过将顶层 RBM 拟合到倒数第二层的后验分布来学习。“下行阶段”（down-pass）从顶层关联记忆的状态开始，使用自上而下的生成连接依次随机激活每个较低层。在下行阶段中，顶层无向连接和生成有向连接保持不变，仅修改自下而上的识别权重。如果在启动下行阶段之前允许关联记忆收敛到其平衡分布，则这等价于 wake-sleep 算法的睡眠阶段。但如果关联记忆由上行阶段初始化，然后在启动下行阶段之前只允许进行少数几次交替吉布斯采样迭代，则这是 wake-sleep 算法的“对比”形式，它消除了从关联记忆的平衡分布中采样的需要。对比形式还修复了睡眠阶段的几个其他问题。它确保识别权重学习到的表示类似于用于真实数据的表示，并且还有助于消除模态平均问题。如果对于某个特定数据向量，当前的识别权重总是选择上一层的某个特定模态，而忽略其他同样能很好地生成数据但差异很大的模态，则下行阶段的学习不会试图改变这些识别权重以恢复任何其他模态，而如果睡眠阶段使用纯祖先传递，则它会尝试这样做。纯祖先传递必须首先通过长时间的吉布斯采样从顶层关联记忆中获得平衡样本。通过使用顶层关联
label may only provide a few bits of constraint on the parameters, so over-fitting is typically more of a problem than under-fitting. Going back and refitting the earlier models may, therefore, cause more harm than good. Unsupervised methods, however, can use very large unlabeled datasets and each case may be very high-dimensional thus providing many bits of constraint on a generative model. Under-fitting is then a serious problem which can be alleviated by a subsequent stage of back-fitting in which the weights that were learned first are revised to fit in better with the weights that were learned later. After greedily learning good initial values for the weights in every layer, we untie the “recognition” weights that are used for inference from the “generative” weights that define the model, but retain the restriction that the posterior in each layer must be approximated by a factorial distribution in which the variables within a layer are conditionally independent given the values of the variables in the layer below. A variant of the wake-sleep algorithm described in Hinton et al. (1995) can then be used to allow the higher-level weights to influence the lower level ones. In the “up-pass”, the recognition weights are used in a bottom-up pass that stochastically picks a state for every hidden variable. The generative weights on the directed connections are then adjusted using the maximum likelihood learning rule in Eq. 2.5. The weights on the undirected connections at the top level are learned as before by fitting the top-level RBM to the posterior distribution of the penultimate layer. The “down-pass” starts with a state of the top-level associative memory and uses the top-down generative connections to stochastically activate each lower layer in turn. During the down-pass, the top-level undirected connections and the generative directed connections are not changed. Only the bottom-up recognition weights are modified. This is equivalent to the sleep phase of the wake-sleep algorithm if the associative memory is allowed to settle to its equilibrium distribution before initiating the down-pass. But if the associative memory is initialized by an up-pass and then only allowed to run for a few iterations of alternating Gibbs sampling before initiating the down-pass, this is a “contrastive” form of the wake-sleep algorithm which eliminates the need to sample from the equilibrium distribution of the associative memory. The contrastive form also fixes several other problems of the sleep phase. It ensures that the recognition weights are being learned for representations that resemble those used for real data and it also helps to eliminate the problem of mode averaging. If, given a particular data vector, the current recognition weights always pick a particular mode at the level above and ignore other very different modes that are equally good at generating the data, the learning in the down-pass will not try to alter those recognition weights to recover any of the other modes as it would if the sleep phase used a pure ancestral pass. A pure ancestral pass would have to start by using prolonged Gibbs sampling to get an equilibrium sample from the top-level associative memory. By using a top-level associa-
图 6：所有 49 个网络猜对但第二个猜测的概率与最佳猜测的概率相差在 0.3 以内的案例。真实类别按标准扫描顺序排列。记忆，我们还消除了唤醒阶段的一个问题：独立的顶层单元似乎是进行祖先传递所必需的，但它们意味着顶层权重的变分近似非常差。附录 B 使用类似 MATLAB 风格的伪代码详细说明了图 1 所示网络的上下算法。为简单起见，没有权重惩罚、没有动量，且所有参数使用相同的学习率。此外，训练数据缩减为单个样本。
Figure 6: All 49 cases in which the network guessed right but had a second guess whose probability was within 0.3 of the probability of the best guess. The true classes are arranged in standard scan order. tive memory we also eliminate a problem in the wake phase: Independent top-level units seem to be required to allow an ancestral pass, but they mean that the variational approximation is very poor for the top layer of weights. Appendix B specifies the details of the up-down algorithm using matlab-style pseudo-code for the network shown in figure 1. For simplicity, there is no penalty on the weights, no momentum, and the same learning rate for all parameters. Also, the training data is reduced to a single case.
MNIST 手写数字数据库包含 60,000 张训练图像和 10,000 张测试图像。这个公开数据集上已经有许多不同模式识别技术的结果发表，因此它是评估新模式识别方法的理想选择。对于 MNIST 学习任务的"基本"版本，不提供几何知识，也没有对训练集进行特殊预处理或增强，因此一个未知但固定的像素随机排列不会影响学习算法。对于这个"置换不变"版本的任务，我们的网络在官方测试集上的泛化性能是 1.25%的错误率。图 1 所示的网络 6 是在 44,000 张训练图像上训练的，这些图像被分为 440 个平衡的小批量，每个小批量包含每个数字类别的 10 个样本。权重在每个小批量之后更新。
The MNIST database of handwritten digits contains 60,000 training images and 10,000 test images. Results for many different pattern recognition techniques are already published for this publicly available database so it is ideal for evaluating new pattern recognition methods. For the “basic” version of the MNIST learning task, no knowledge of geometry is provided and there is no special pre-processing or enhancement of the training set, so an unknown but fixed random permutation of the pixels would not affect the learning algorithm. For this “permutation-invariant” version of the task, the generalization performance of our network was 1.25% errors on the official test set. The network 6 shown in figure 1 was trained on 44,000 of the training images that were divided into 440 balanced mini-batches each containing 10 examples of each digit class. The weights were updated after each mini-batch.
图 7：网络出错的 125 个测试案例。每个案例由网络的猜测标记。真实类别按标准扫描顺序排列。在训练的初始阶段，使用第 4 节描述的贪心算法，从底层开始分别训练每一层权重。每层在训练集上训练 30 次遍历（称为"epochs"）。训练期间，每个 RBM 的"可见"层中的单元具有 0 到 1 之间的实值活动。这些是在学习底层权重时的归一化像素强度。为了训练更高层的权重，RBM 中可见单元的实值活动是低层 RBM 中隐藏单元的激活概率。每个 RBM 的隐藏层在训练该 RBM 时使用随机二值。贪心训练在 3GHz Xeon 处理器上的 Matlab 中每层耗时数小时，完成后测试集的错误率为 2.49%（关于网络测试的详细信息见下文）。训练顶层权重（联想记忆中的权重）时，标签作为输入的一部分提供。标签通过打开 10 个单元的"softmax"组中的一个单元来表示。当从上层活动重构该组中的活动时，只允许一个单元活跃，选择单元 i 的概率由下式给出：
Figure 7: The 125 test cases that the network got wrong. Each case is labeled by the network’s guess. The true classes are arranged in standard scan order. In the initial phase of training, the greedy algorithm described in section 4 was used to train each layer of weights separately, starting at the bottom. Each layer was trained for 30 sweeps through the training set (called “epochs”). During training, the units in the “visible” layer of each RBM had real-valued activities between 0 and 1. These were the normalized pixel intensities when learning the bottom layer of weights. For training higher layers of weights, the real-valued activities of the visible units in the RBM were the activation probabilities of the hidden units in the lower-level RBM. The hidden layer of each RBM used stochastic binary values when that RBM was being trained. The greedy training took a few hours per layer in Matlab on a 3GHz Xeon processor and when it was done, the error-rate on the test set was 2.49% (see below for details of how the network is tested). When training the top layer of weights (the ones in the associative memory) the labels were provided as part of the input. The labels were represented by turning on one unit in a “softmax” group of 10 units. When the activities in this group were reconstructed from the activities in the layer above, exactly one unit was allowed to be active and the probability of picking unit i was given by:
\[ \exp(x_j) \qquad (10) \] 其中\(x_i\)是单元 i 接收的总输入。奇怪的是，学习规则不受 softmax 组内单元之间竞争的影响，因此突触不需要知道哪个单元在与哪个其他单元竞争。竞争影响单元激活的概率，但只有这个概率影响学习。在逐层贪心训练之后，网络使用第 5 节描述的上下算法，以不同的学习率和权重衰减训练了 300 个 epochs。学习率、动量以及权重衰减通过多次训练网络并在从完整训练集剩余部分中取出的 10,000 张图像的单独验证集上观察其表现来选择。在上下算法的前 100 个 epochs 中，上行传递之后在联想记忆中进行三次完整的交替吉布斯采样迭代，然后进行下行传递。在接下来的 100 个 epochs 中，进行了六次迭代，最后 100 个 epochs 中，进行了十次迭代。每次吉布斯采样迭代次数增加时，验证集上的错误率显著下降。在验证集上表现最好的网络随后进行测试，其错误率为 1.39%。然后该网络在所有 60,000 张训练图像上继续训练，直到其在完整训练集上的错误率与之前在 44,000 张图像初始训练集上的最终错误率一样低。这又花费了 59 个 epochs，使总学习时间约为一周。最终网络的错误率为 1.25%。网络出错的案例显示在图 7 中。网络正确但第二最佳概率与最佳概率相差 0.3 以内的 49 个案例显示在图 6 中。1.25%的错误率与使用一个或两个隐藏层并通过反向传播算法训练以优化区分的前馈神经网络所达到的错误率相比非常有利（见表 1，位于参考文献之后）。当这些网络的详细连接不是为此任务手工设计时，
\[ \exp(x_j) \qquad (10) \] where \(x_i\) is the total input received by unit i. Curiously, the learning rules are unaffected by the competition between units in a softmax group, so the synapses do not need to know which unit is competing with which other unit. The competition affects the probability of a unit turning on, but it is only this probability that affects the learning. After the greedy layer-by-layer training, the network was trained, with a different learning rate and weight-decay, for 300 epochs using the up-down algorithm described in section 5. The learning rate, momentum, and weight-decay were chosen by training the network several times and observing its performance on a separate validation set of 10,000 images that were taken from the remainder of the full training set. For the first 100 epochs of the up-down algorithm, the up-pass was followed by three full iterations of alternating Gibbs sampling in the associative memory before performing the down-pass. For the second 100 epochs, six iterations were performed, and for the last 100 epochs, ten iterations were performed. Each time the number of iterations of Gibbs sampling was raised, the error on the validation set decreased noticeably. The network that performed best on the validation set was then tested and had an error rate of 1.39%. This network was then trained on all 60,000 training images until its error-rate on the full training set was as low as its final error-rate had been on the initial training set of 44,000 images. This took a further 59 epochs making the total learning time about a week. The final network had an error-rate of 1.25%. The errors made by the network are shown in figure 7. The 49 cases that the network gets correct but for which the second best probability is within 0.3 of the best probability are shown in figure 6. The error-rate of 1.25% compares very favorably with the error-rates achieved by feed-forward neural networks that have one or two hidden layers and are trained to optimize discrimination using the back-propagation algorithm (see table 1, appearing after the references). When the detailed connectivity of these networks is not hand-crafted for this
测试网络的一种方法是使用从图像开始的随机上行传递来固定关联记忆下层 500 个单元的二元状态。在这些状态固定后，标签单元被赋予初始实值激活值
One way to test the network is to use a stochastic up-pass from the image to fix the binary states of the 500 units in the lower layer of the associative memory. With these states fixed, the label units are given initial real-valued activities of
0.1，然后进行几次交替吉布斯采样以激活正确的标签单元。这种测试方法得到的错误率比上述报告的结果高出近 1%。更好的方法是先固定关联记忆下层 500 个单元的二元状态，然后依次开启每个标签单元，并计算所得 510 分量二元向量的精确自由能。图 8：每一行显示在特定标签钳制下从生成模型中抽取的 10 个样本。顶层关联记忆在样本之间运行 1000 次交替吉布斯采样。几乎所有所需的计算都与开启哪个标签单元无关（Teh 和 Hinton, 2001），并且该方法计算标签上的精确条件均衡分布，而不是像前一种方法那样通过吉布斯采样近似。该方法给出的错误率比引用的结果高出约 0.5%，原因是上行传递中的随机决策。我们可以通过两种方式消除这种噪声。最简单的方法是使用激活概率代替随机二元状态使上行传递变为确定性的。第二种方法是重复随机上行传递二十次，并在选择最佳标签之前对二十次重复中的标签概率或标签对数概率进行平均。这两种平均方法给出的结果几乎相同，并且这些结果也与使用确定性上行传递非常相似，后者正是报告结果所采用的方法。
0.1 and a few iterations of alternating Gibbs sampling are then used to activate the correct label unit. This method of testing gives error rates that are almost 1% higher than the rates reported above. A better method is to first fix the binary states of the 500 units in the lower layer of the associative memory and to then turn on each of the label units in turn and compute the exact free energy of the resulting 510 component binary vector. Figure 8: Each row shows 10 samples from the generative model with a particular label clamped on. The top-level associative memory is run for 1000 iterations of alternating Gibbs sampling between samples. Almost all the computation required is independent of which label unit is turned on (Teh and Hinton, 2001) and this method computes the exact conditional equilibrium distribution over labels instead of approximating it by Gibbs sampling which is what the previous method is doing. This method gives error rates that are about 0.5% higher than the ones quoted because of the stochastic decisions made in the up-pass. We can remove this noise in two ways. The simplest is to make the up-pass deterministic by using probabilities of activation in place of stochastic binary states. The second is to repeat the stochastic up-pass twenty times and average either the label probabilities or the label log probabilities over the twenty repetitions before picking the best one. The two types of average give almost identical results and these results are also very similar to using a deterministic up-pass, which was the method used for the reported results.
为了从模型生成样本，我们在顶层联想记忆中进行交替吉布斯采样，直到马尔可夫链收敛到平衡分布。然后，我们使用该分布的一个样本作为下面几层的输入，并通过生成连接进行一次下行传播来生成图像。如果在吉布斯采样期间将标签单元固定到特定类别，我们可以看到模型类条件分布的图像。图 8 显示了每个类别的一系列图像，这些图像通过在样本之间进行 1000 次吉布斯迭代生成。我们还可以通过提供一个随机二值图像作为输入来初始化顶层两层的状态。图 9 展示了当允许联想记忆自由运行但标签固定时，其类条件状态如何演变。
To generate samples from the model, we perform alternating Gibbs sampling in the top-level associative memory until the Markov chain converges to the equilibrium distribution. Then we use a sample from this distribution as input to the layers below and generate an image by a single down-pass through the generative connections. If we clamp the label units to a particular class during the Gibbs sampling we can see images from the model’s class-conditional distributions. Figure 8 shows a sequence of images for each class that were generated by allowing 1000 iterations of Gibbs sampling between samples. We can also initialize the state of the top two layers by providing a random binary image as input. Figure 9 shows how the class-conditional state of the associative memory then evolves when it is allowed to run freely, but with the
图 9：每行显示生成模型的 10 个样本，其中固定了一个特定标签。顶层联想记忆通过从随机二值图像（每个像素以 0.5 的概率开启）进行一次上行传播来初始化。
Figure 9: Each row shows 10 samples from the generative model with a particular label clamped on. The top-level associative memory is initialized by an up-pass from a random binary image in which each pixel is on with a probability of
第一列显示从该初始高层状态进行下行传播的结果。后续列通过在联想记忆中进行 20 次交替吉布斯采样产生。标签固定。这种内部状态通过每 20 次迭代执行一次下行传播来“观察”，以了解联想记忆的思维。此处使用“思维”一词并非隐喻。我们认为，心理状态是一个假设的外部世界的状态，在该世界中，高层内部表征将构成真实的感知。该图展示的正是那个假设的世界。
0.5. The first column shows the results of a down-pass from this initial high-level state. Subsequent columns are produced by 20 iterations of alternating Gibbs sampling in the associative memory. label clamped. This internal state is “observed” by performing a down-pass every 20 iterations to see what the associative memory has in mind. This use of the word “mind” is not intended to be metaphorical. We believe that a mental state is the state of a hypothetical, external world in which a high-level internal representation would constitute veridical perception. That hypothetical world is what the figure shows.
我们已经证明，可以一次一层地学习一个深度、密集连接的信念网络。实现这一点的直观方法是假设在学习低层时高层不存在，但这与使用简单的因子分解近似来替代难解的后验分布不相容。为了使这些近似有效，我们需要真实后验尽可能接近因子分解形式。因此，我们不忽略高层，而是假设它们存在但具有捆绑的权重，这些权重受约束以实现一个互补先验，使得真实后验恰好是因子分解的。这等价于一个无向模型，可以通过对比散度高效学习。它也可以被视为约束变分学习，因为一个惩罚项——近似后验与真实后验之间的散度——已被替换为先验必须使变分近似精确的约束。在每一层学习完成后，其权重与更高层的权重解绑。随着这些高层权重的变化，低层的先验不再互补，因此低层的真实后验分布不再是因子分解的，使用生成权重的转置进行推断也不再正确。尽管如此，我们可以使用变分界来证明调整高层权重能够改进整体生成模型。为了展示我们快速贪婪学习算法的威力，我们用它来初始化一个慢得多的微调算法的权重，该算法学习了一个出色的数字图像及其标签的生成模型。目前尚不清楚这是否是利用快速贪婪算法的最佳方式。或许更好的做法是省略微调，利用贪婪算法的速度来学习一个更大、更深的网络集成或一个更大的训练集。图 1 中的网络拥有的参数数量大约相当于 0.002 立方毫米小鼠皮层（Horace Barlow，私人通信），而数百个如此复杂度的网络可以装进高分辨率 fMRI 扫描的一个单个体素中。这表明可能需要更大的网络才能与人类的形状识别能力相竞争。我们当前的生成模型在许多方面存在局限（Lee and Mumford, 2003）。它专为那些非二进制值可被视为概率的图像而设计（自然图像并非如此）；它在感知过程中自上而下反馈的使用仅限于顶层两层的联想记忆；它没有系统的方法来处理感知不变性；它假设分割已经完成，并且在判别困难时不学习顺序关注物体信息量最大的部分。然而，它确实展示了生成模型相比于判别模型的一些主要优势：1. 生成模型可以在不需要标签反馈的情况下学习低层特征，并且可以学习比判别模型多得多的参数而不会过拟合。在判别学习中，每个训练样本只通过指定标签所需的信息比特数来约束参数。对于生成模型，每个训练样本通过指定输入所需的比特数来约束参数。2. 通过从模型中生成，可以容易地看出网络学习到了什么。3. 可以通过从深层隐藏层生成图像来解释其中的非线性、分布式表示。4. 判别学习方法优越的分类性能仅适用于无法学习到好的生成模型的领域。而这一领域集合正随着摩尔定律而不断缩小。
We have shown that it is possible to learn a deep, densely-connected, belief network one layer at a time. The obvious way to do this is to assume that the higher layers do not exist when learning the lower layers, but this is not compatible with the use of simple factorial approximations to replace the intractable posterior distribution. For these approximations to work well, we need the true posterior to be as close to factorial as possible. So instead of ignoring the higher layers, we assume that they exist but have tied weights which are constrained to implement a complementary prior that makes the true posterior exactly factorial. This is equivalent to having an undirected model which can be learned efficiently using contrastive divergence. It can also be viewed as constrained variational learning because a penalty term – the divergence between the approximate and true posteriors – has been replaced by the constraint that the prior must make the variational approximation exact. After each layer has been learned, its weights are untied from the weights in higher layers. As these higher-level weights change, the priors for lower layers cease to be complementary, so the true posterior distributions in lower layers are no longer factorial and the use of the transpose of the generative weights for inference is no longer correct. Nevertheless, we can use a variational bound to show that adapting the higher-level weights improves the overall generative model. To demonstrate the power of our fast, greedy learning algorithm, we used it to initialize the weights for a much slower fine-tuning algorithm that learns an excellent generative model of digit images and their labels. It is not clear that this is the best way to use the fast, greedy algorithm. It might be better to omit the fine-tuning and use the speed of the greedy algorithm to learn an ensemble of larger, deeper networks or a much larger training set. The network in figure 1 has about as many parameters as 0.002 cubic millimeters of mouse cortex (Horace Barlow, pers. comm.), and several hundred networks of this complexity could fit within a single voxel of a high resolution fMRI scan. This suggests that much bigger networks may be required to compete with human shape recognition abilities. Our current generative model is limited in many ways (Lee and Mumford, 2003). It is designed for images in which non-binary values can be treated as probabilities (which is not the case for natural images); its use of top-down feedback during perception is limited to the associative memory in the top two layers; it does not have a systematic way of dealing with perceptual invariances; it assumes that segmentation has already been performed and it does not learn to sequentially attend to the most informative parts of objects when discrimination is difficult. It does, however, illustrate some of the major advantages of generative models as compared to discriminative ones: 1. Generative models can learn low-level features without requiring feedback from the label and they can learn many more parameters than discriminative models without overfitting. In discriminative learning, each training case only constrains the parameters by as many bits of information as are required to specify the label. For a generative model, each training case constrains the parameters by the number of bits required to specify the input. 2. It is easy to see what the network has learned by generating from its model. 3. It is possible to interpret the non-linear, distributed representations in the deep hidden layers by generating images from them. 4. The superior classification performance of discriminative learning methods only holds for domains in which it is not possible to learn a good generative model. This set of domains is being eroded by Moore's law.