一种深度信念网络的快速学习算法

A Fast Learning Algorithm for Deep Belief Nets

杰弗里·辛顿 Geoffrey Hinton · Neural Computation (2006) · 2006-07-01 · 2006 ↗

摘要 · Abstract

多伦多大学计算机科学系，国王学院路 10 号，多伦多，加拿大 M5S 3G4

Department of Computer Science University of Toronto 10 Kings College Road Toronto, Canada M5S 3G4

提出互补先验消除解释消除效应，实现深度信念网的高效推理。
Proposes complementary priors to eliminate explaining away, enabling efficient inference in deep belief nets.
推导出快速贪心逐层预训练算法，通过对比散度一次学习一层。
Derives a fast greedy layer-wise pretraining algorithm that learns one layer at a time via contrastive divergence.
引入用于微调的上下算法，是唤醒-睡眠的对比变体，避免模式平均。
Introduces the up-down algorithm for fine-tuning, a contrastive variant of wake-sleep that avoids mode averaging.
在 MNIST 上达到 1.25%测试错误率，优于 SVM 和反向传播等判别方法。
Achieves 1.25% test error on MNIST, outperforming discriminative methods like SVM and backpropagation.
通过从顶层联想记忆状态生成图像，展示可解释性。
Demonstrates interpretability by generating images from the top-level associative memory's states.
表明生成模型相比判别模型可学习更多参数而不过拟合。
Shows that generative models can learn more parameters without overfitting compared to discriminative models.

模型假设二元随机单元，限制了在非二元或连续数据上的应用。
Model assumes binary stochastic units, limiting applicability to non-binary or continuous data.
生成模型将像素强度视为概率，不适用于自然图像中的非概率值。
Generative model treats pixel intensities as probabilities, unsuitable for natural images with non-probabilistic values.
感知中的自上而下反馈局限于联想记忆，而非完整分层推理。
Top-down feedback during perception is confined to the associative memory, not full hierarchical inference.
缺乏处理平移或旋转等感知不变性的系统机制。
No systematic mechanism for handling perceptual invariances like translation or rotation.
假设事先完成分割，不学习难以区分情况下的顺序注意。
Assumes segmentation is performed beforehand and does not learn sequential attention for difficult discriminations.

杰弗里·E·辛顿和西蒙·奥辛德罗 Geoffrey E. Hinton and Simon Osindero
作者信息 Yee-Whye Teh
摘要 Abstract
1 引言 1 Introduction
这可能是另一条感觉通路的顶层 This could be the top level of another sensory pathway
2 互补先验 2 Complementary priors
2.1 带权值共享的无限有向模型 2.1 An infinite directed model with tied weights
3 受限玻尔兹曼机与对比散度学习 3 Restricted Boltzmann machines and contrastive divergence learning
WW WW
对比散度学习 >< ∞∞ ji hv
4 一种用于转换表示的贪婪学习算法 4 A greedy learning algorithm for transforming representations
5 使用上下算法的反向拟合 5 Back-Fitting with the up-down algorithm
6.1 训练网络 6.1 Training the network
6.2 测试网络 6.2 Testing the network
7 窥探神经网络的思维 7 Looking into the mind of a neural network
8 结论 8 Conclusion
论文插图 Figures