使用深度卷积神经网络的 ImageNet 分类

摘要 Abstract

我们训练了一个大型深度卷积神经网络，将 ImageNet LSVRC-2010 竞赛中的 120 万张高分辨率图像分类为 1000 个不同的类别。在测试数据上，我们取得了 top-1 和 top-5 错误率分别为 37.5% 和 17.0%，这显著优于之前的最优结果。该神经网络拥有 6000 万个参数和 65 万个神经元，由五个卷积层（其中一些后接最大池化层）和三个全连接层组成，最后是一个 1000 路的 softmax。为了加快训练速度，我们使用了非饱和神经元以及非常高效的 GPU 卷积运算实现。为了减少全连接层中的过拟合，我们采用了一种最近开发的称为“dropout”的正则化方法，该方法被证明非常有效。我们还将该模型的一个变体参加了 ILSVRC-2012 竞赛，并取得了获胜的 top-5 测试错误率 15.3%，而第二名参赛者的错误率为 26.2%。

We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.

1 引言 1 Introduction

当前的目标识别方法主要使用机器学习方法。为了提高性能，我们可以收集更大的数据集、学习更强大的模型以及使用更好的防止过拟合技术。直到最近，标记图像数据集还相对较小——大约数万张图像（例如，NORB [16]、Caltech-101/256 [8, 9] 和 CIFAR-10/100 [12]）。对于这种规模的数据集，简单的识别任务可以很好地解决，特别是如果它们通过保留标签的变换进行增强。例如，当前 MNIST 数字识别任务的最佳错误率（<0.3%）接近人类水平 [4]。但是，现实环境中的物体表现出相当大的变异性，因此要学习识别它们，必须使用更大的训练集。事实上，小图像数据集的缺点已被广泛认识到（例如，Pinto 等人 [21]），但直到最近才有可能收集到数百万张图像的标记数据集。新的更大数据集包括 LabelMe [23]（包含数十万张完全分割的图像）和 ImageNet [6]（包含超过 22000 个类别的超过 1500 万张标记的高分辨率图像）。要从数百万张图像中学习数千个物体，我们需要一个具有大学习容量的模型。然而，物体识别任务的巨大复杂性意味着即使像 ImageNet 这样大的数据集也无法完全描述此问题，因此我们的模型还应该具有大量的先验知识来补偿我们没有的数据。卷积神经网络（CNN）就是这类模型之一 [16, 11, 13, 18, 15, 22, 26]。它们的容量可以通过改变深度和宽度来控制，并且它们对图像的本质（即统计平稳性和像素依赖关系的局部性）做出了强有力且基本正确的假设。因此，与具有相似大小层的标准前馈神经网络相比，CNN 的连接和参数少得多，因此更容易训练，而它们的理论最佳性能可能只差一点点。尽管 CNN 具有吸引人的特性，并且其局部架构相对高效，但它们在大规模应用于高分辨率图像时仍然过于昂贵。幸运的是，当前的 GPU 配合高度优化的二维卷积实现，足够强大以支持训练较大规模的 CNN，并且像 ImageNet 这样的近期数据集包含了足够的标记样本来训练此类模型而不会出现严重的过拟合。本文的具体贡献如下：我们在 ILSVRC-2010 和 ILSVRC-2012 竞赛 [2] 中使用的 ImageNet 子集上训练了迄今为止最大的卷积神经网络之一，并取得了这些数据集上迄今为止最好的结果。我们编写了一个高度优化的 GPU 实现，用于二维卷积以及训练卷积神经网络固有的所有其他操作，并将其公开发布。我们的网络包含许多新颖且不寻常的特性，这些特性提高了其性能并减少了训练时间，详见第 3 节。我们网络的规模使得过拟合成为一个严重问题，即使有 120 万个标记训练样本也是如此，因此我们使用了多种有效的防止过拟合技术，详见第 4 节。我们的最终网络包含五个卷积层和三个全连接层，这种深度似乎很重要：我们发现移除任何卷积层（每个卷积层包含的模型参数不超过 1%）都会导致性能下降。最终，网络的规模主要受限于当前 GPU 上的内存容量以及我们愿意承受的训练时间。我们的网络在两块 GTX 580 3GB GPU 上训练需要五到六天。我们所有的实验表明，只需等待更快的 GPU 和更大的数据集，我们的结果就可以得到改进。

Current approaches to object recognition make essential use of machine learning methods. To improve their performance, we can collect larger datasets, learn more powerful models, and use better techniques for preventing overfitting. Until recently, datasets of labeled images were relatively small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and CIFAR-10/100 [12]). Simple recognition tasks can be solved quite well with datasets of this size, especially if they are augmented with label-preserving transformations. For example, the current-best error rate on the MNIST digit-recognition task (<0.3%) approaches human performance [4]. But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets. And indeed, the shortcomings of small image datasets have been widely recognized (e.g., Pinto et al. [21]), but it has only recently become possible to collect labeled datasets with millions of images. The new larger datasets include LabelMe [23], which consists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of over 15 million labeled high-resolution images in over 22,000 categories. To learn about thousands of objects from millions of images, we need a model with a large learning capacity. However, the immense complexity of the object recognition task means that this problem cannot be specified even by a dataset as large as ImageNet, so our model should also have lots of prior knowledge to compensate for all the data we don’t have. Convolutional neural networks (CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse. Despite the attractive qualities of CNNs, and despite the relative efficiency of their local architecture, they have still been prohibitively expensive to apply in large scale to high-resolution images. Luckily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet contain enough labeled examples to train such models without severe overfitting. The specific contributions of this paper are as follows: we trained one of the largest convolutional neural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012 competitions [2] and achieved by far the best results ever reported on these datasets. We wrote a highly-optimized GPU implementation of 2D convolution and all the other operations inherent in training convolutional neural networks, which we make available publicly. Our network contains a number of new and unusual features which improve its performance and reduce its training time, which are detailed in Section 3. The size of our network made overfitting a significant problem, even with 1.2 million labeled training examples, so we used several effective techniques for preventing overfitting, which are described in Section 4. Our final network contains five convolutional and three fully-connected layers, and this depth seems to be important: we found that removing any convolutional layer (each of which contains no more than 1% of the model’s parameters) resulted in inferior performance. In the end, the network’s size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate. Our network takes between five and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available.

2 数据集 2 The Dataset

ImageNet 是一个包含超过 1500 万张标注高分辨率图像的数据集，涵盖大约 22,000 个类别。这些图像从网络收集，并由人类标注者使用亚马逊的 Mechanical Turk 众包工具进行标注。自 2010 年起，作为 Pascal 视觉对象挑战的一部分，每年举行一次名为 ImageNet 大规模视觉识别挑战赛（ILSVRC）的竞赛。ILSVRC 使用 ImageNet 的一个子集，包含 1000 个类别，每个类别大约 1000 张图像。总共有大约 120 万张训练图像、5 万张验证图像和 15 万张测试图像。ILSVRC-2010 是唯一公开测试集标签的 ILSVRC 版本，因此我们大部分实验都在该版本上进行。由于我们也参加了 ILSVRC-2012 竞赛，在第 6 节中我们也会报告在该版本数据集上的结果，但该版本的测试集标签不可获取。在 ImageNet 上，通常报告两种错误率：top-1 和 top-5，其中 top-5 错误率是指正确标签不在模型认为最可能的五个标签之中的测试图像比例。ImageNet 由分辨率可变的图像组成，而我们的系统需要恒定的输入维度。因此，我们将图像下采样到固定的 256×256 分辨率。给定一张矩形图像，我们首先将图像缩放至较短边长度为 256，然后从所得图像中裁剪出中央 256×256 的块。我们没有对图像进行任何其他预处理，只是从每个像素中减去训练集上的平均活动值。因此，我们是在（中心化的）原始 RGB 像素值上训练我们的网络。

ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories. The images were collected from the web and labeled by human labelers using Amazon’s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of 1000 categories. In all, there are roughly 1.2 million training images, 50,000 validation images, and 150,000 testing images. ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is the version on which we performed most of our experiments. Since we also entered our model in the ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as well, for which test set labels are unavailable. On ImageNet, it is customary to report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the model. ImageNet consists of variable-resolution images, while our system requires a constant input dimensionality. Therefore, we down-sampled the images to a fixed resolution of 256 × 256. Given a rectangular image, we first rescaled the image such that the shorter side was of length 256, and then cropped out the central 256 × 256 patch from the resulting image. We did not pre-process the images in any other way, except for subtracting the mean activity over the training set from each pixel. So we trained our network on the (centered) raw RGB values of the pixels.

3 架构 3 The Architecture

我们的网络架构总结在图 2 中。它包含八个学习层——五个卷积层和三个全连接层。下面，我们描述网络架构中一些新颖或不寻常的特征。第 3.1-3.4 节按照我们估计的重要性排序，最重要的在前。

The architecture of our network is summarized in Figure 2. It contains eight learned layers — five convolutional and three fully-connected. Below, we describe some of the novel or unusual features of our network's architecture. Sections 3.1-3.4 are sorted according to our estimation of their importance, with the most important first.

图 1：一个带有 ReLU（实线）的四层卷积神经网络在 CIFAR-10 上达到 25%的训练错误率，比具有 tanh 神经元的等效网络快六倍。

Figure 1: A four-layer convolutional neural network with ReLUs (solid line) reaches a 25% training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons

建模神经元输出 f 作为输入 x 的函数的标准方法是\(f(x) = \tanh(x)\)或\(f(x) = (1 + e^{-x})^{-1}\)。在使用梯度下降的训练时间方面，这些饱和非线性函数比非饱和非线性函数\(f(x) = \max(0, x)\)慢得多。按照 Nair 和 Hinton [20]的说法，我们将具有这种非线性的神经元称为整流线性单元（ReLU）。带有 ReLU 的深度卷积神经网络比具有 tanh 单元的等效网络训练速度快几倍。这在图 1 中得到了证明，该图显示了在 CIFAR-10 数据集上，一个特定的四层卷积网络达到 25%训练错误率所需的迭代次数。该图表明，如果我们使用传统的饱和神经元模型，我们将无法在这项工作中实验如此大的神经网络。我们并不是第一个考虑 CNN 中传统神经元模型替代方案的人。例如，Jarrett 等人[11]声称非线性函数\(f(x) = |\tanh(x)|\)

The standard way to model a neuron's output f as a function of its input x is with \(f(x) = \tanh(x)\) or \(f(x) = (1 + e^{-x})^{-1}\). In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity \(f(x) = \max(0, x)\). Following Nair and Hinton [20], we refer to neurons with this nonlinearity as Rectified Linear Units (ReLUs). Deep convolutional neural networks with ReLUs train several times faster than their equivalents with tanh units. This is demonstrated in Figure 1, which shows the number of iterations required to reach 25% training error on the CIFAR-10 dataset for a particular four-layer convolutional network. This plot shows that we would not have been able to experiment with such large neural networks for this work if we had used traditional saturating neuron models. We are not the first to consider alternatives to traditional neuron models in CNNs. For example, Jarrett et al. [11] claim that the nonlinearity \(f(x) = |\tanh(x)|\)

The architecture of our network is summarized in Figure 2. It contains eight learned layers — five convolutional and three fully-connected. Below, we describe some of the novel or unusual features of our network’s architecture. Sections 3.1-3.4 are sorted according to our estimation of their importance, with the most important first.

Figure 1: A four-layer convolutional neural network with ReLUs (solid line) reaches a 25% training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons

该方法与他们在 Caltech-101 数据集上使用的对比度归一化后接局部平均池化方法配合得特别好。然而，在该数据集上主要关注的是防止过拟合，因此他们观察到的效果与我们使用 ReLU 时报告的加速拟合训练集的能力不同。更快的学习对在大型数据集上训练的大型模型的性能有显著影响。

works particularly well with their type of contrast normalization followed by local average pooling on the Caltech-101 dataset. However, on this dataset the primary concern is preventing overfitting, so the effect they are observing is different from the accelerated ability to fit the training set which we report when using ReLUs. Faster learning has a great influence on the performance of large models trained on large datasets.

单块 GTX 580 GPU 仅有 3GB 内存，这限制了可训练网络的最大规模。事实表明，120 万训练样本足以训练那些太大而无法放在单块 GPU 上的网络。因此，我们将网络分布在两块 GPU 上。当前的 GPU 特别适合跨 GPU 并行化，因为它们能够直接读写彼此的内存，而无需经过主机内存。我们采用的并行化方案本质上将一半的核（或神经元）放在每个 GPU 上，并有一个额外技巧：GPU 仅在特定层进行通信。这意味着，例如，第 3 层的核从第 2 层的所有核图中接收输入。然而，第 4 层的核仅从位于同一 GPU 的第 3 层的核图中接收输入。选择连接模式是一个交叉验证问题，但这使我们能够精确调整通信量，直到其占计算量的可接受比例。最终架构与 Cireșan 等人[5]使用的“柱状”CNN 有些相似，不同之处在于我们的柱不是独立的（见图 2）。与每个卷积层核数减半的单 GPU 网络相比，该方案使我们的 top-1 和 top-5 错误率分别降低了 1.7%和 1.2%。双 GPU 网络的训练时间略少于单 GPU 网络 2。

A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU. Therefore we spread the net across two GPUs. Current GPUs are particularly well-suited to cross-GPU parallelization, as they are able to read from and write to one another's memory directly, without going through host machine memory. The parallelization scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers. This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on the same GPU. Choosing the pattern of connectivity is a problem for cross-validation, but this allows us to precisely tune the amount of communication until it is an acceptable fraction of the amount of computation. The resultant architecture is somewhat similar to that of the "columnar" CNN employed by Cire¸ san et al. [5], except that our columns are not independent (see Figure 2). This scheme reduces our top-1 and top-5 error rates by 1.7% and 1.2%, respectively, as compared with a net with half as many kernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time to train than the one-GPU net 2.

ReLU 具有一个理想的特性：它们不需要输入归一化来防止饱和。如果至少有一些训练样本产生对 ReLU 的正输入，那么该神经元就会学习。然而，我们仍然发现以下局部归一化方案有助于泛化。用\(a_{i}^{x,y}\)表示通过在第\(i\)个核在位置\(x, y\)上应用 ReLU 非线性后计算的神经元活动值，则响应归一化后的活动值\(b_{i}^{x,y}\)由表达式给出

ReLUs have the desirable property that they do not require input normalization to prevent them from saturating. If at least some training examples produce a positive input to a ReLU, learning will happen in that neuron. However, we still find that the following local normalization scheme aids generalization. Denoting by \(a_{i}^{x,y}\) the activity of a neuron computed by applying kernel i at position \(x, y\) and then applying the ReLU nonlinearity, the response-normalized activity \(b_{i}^{x,y}\) is given by the expression

其中求和运行在同一空间位置上的\(n\)个“相邻”核图上，\(N\)是该层核的总数。核图的顺序当然是任意的，并在训练开始前确定。这种响应归一化实现了一种侧抑制形式，其灵感来自真实神经元中的类型，在不同核计算的神经元输出之间为大活动创建竞争。常数\(k\)、\(n\)、\(\alpha\)和\(\beta\)是超参数，其值通过验证集确定；我们使用\(k=2\)、\(n=5\)、\(\alpha=10^{-4}\)和\(\beta=0.75\)。我们在某些层应用 ReLU 非线性后应用了此归一化（见第 3.5 节）。该方案与 Jarrett 等人[11]的局部对比度归一化方案有些相似，但我们的方案更准确地说应称为“亮度归一化”，因为我们没有减去平均活动。响应归一化使我们的 top-1 和 top-5 错误率分别降低了 1.4%和 1.2%。我们还在 CIFAR-10 数据集上验证了该方案的有效性：一个四层 CNN 在未归一化时测试错误率为 13%，在归一化后为 11%3。

where the sum runs over \(n\) "adjacent" kernel maps at the same spatial position, and \(N\) is the total number of kernels in the layer. The ordering of the kernel maps is of course arbitrary and determined before training begins. This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels. The constants \(k\), \(n\), \(\alpha\), and \(\beta\) are hyper-parameters whose values are determined using a validation set; we used \(k=2\), \(n=5\), \(\alpha=10^{-4}\), and \(\beta=0.75\). We applied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5). This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al. [11], but ours would be more correctly termed "brightness normalization", since we do not subtract the mean activity. Response normalization reduces our top-1 and top-5 error rates by 1.4% and 1.2%, respectively. We also verified the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer CNN achieved a 13% test error rate without normalization and 11% with normalization 3.

CNN 中的池化层汇总同一核映射中邻近神经元组的输出。传统上，相邻池化单元汇总的邻域不重叠（例如[17, 11, 4]）。更准确地说，池化层可以看作由间距为 s 个像素的池化单元网格构成，每个单元汇总以其位置为中心、大小为 z×z 的邻域。如果设 s=z，则得到 CNN 中常用的传统局部池化。如果设 s<z，则得到重叠池化。我们在整个网络中使用重叠池化，取 s=2, z=3。与产生同等维度输出的非重叠方案（s=2, z=2）相比，该方案将 top-1 和 top-5 错误率分别降低了 0.4%和 0.3%。我们在训练中普遍观察到，采用重叠池化的模型稍微更难以过拟合。

Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., [17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced \(s\) pixels apart, each summarizing a neighborhood of size \(z \times z\) centered at the location of the pooling unit. If we set \(s = z\), we obtain traditional local pooling as commonly employed in CNNs. If we set \(s < z\), we obtain overlapping pooling. This is what we use throughout our network, with \(s = 2\) and \(z = 3\). This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively, as compared with the non-overlapping scheme \(s = 2, z = 2\), which produces output of equivalent dimensions. We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit.

现在我们来描述我们 CNN 的整体架构。如图 2 所示，网络包含八个带权重的层；前五个是卷积层，后三个是全连接层。最后一个全连接层的输出被送入一个 1000 维的 softmax，产生一个在 1000 个类别标签上的分布。我们的网络最大化多项逻辑回归目标，这等价于最大化训练样本上预测分布下正确标签的对数概率的平均值。第二、第四、第五卷积层的核仅与上一层中位于同一 GPU 上的核映射相连（见图 2）。第三卷积层的核与第二层的所有核映射相连。全连接层中的神经元与上一层中的所有神经元相连。第一和第二卷积层之后是响应归一化层。响应归一化层以及第五卷积层之后是最大池化层，类型如第 3.4 节所述。在每个卷积层和全连接层的输出上应用 ReLU 非线性。第一个卷积层用 96 个大小为 11×11×3 的核对 224×224×3 的输入图像进行滤波。

Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net contains eight layers with weights; the first five are convolutional and the remaining three are fully-connected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels. Our network maximizes the multinomial logistic regression objective, which is equivalent to maximizing the average across training cases of the log-probability of the correct label under the prediction distribution. The kernels of the second, fourth, and fifth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU (see Figure 2). The kernels of the third convolutional layer are connected to all kernel maps in the second layer. The neurons in the fully-connected layers are connected to all neurons in the previous layer. Response-normalization layers follow the first and second convolutional layers. Max-pooling layers, of the kind described in Section 3.4, follow both response-normalization layers as well as the fifth convolutional layer. The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer. The first convolutional layer filters the \(224 \times 224 \times 3\) input image with 96 kernels of size \(11 \times 11 \times 3\)

步长为 4 个像素（这是相邻核映射中感受野中心之间的距离

with a stride of 4 pixels (this is the distance between the receptive field centers of neighboring

图 2：我们 CNN 架构的图示，明确展示了两个 GPU 之间的职责划分。一个 GPU 运行图中上部的层部分，另一个 GPU 运行图中下部的层部分。GPU 仅在特定层通信。网络的输入是 150,528 维，网络其余层中的神经元数量依次为：253,440–186,624–64,896–64,896–43,264–4096–4096–1000。

Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities between the two GPUs. One GPU runs the layer-parts at the top of the figure while the other runs the layer-parts at the bottom. The GPUs communicate only at certain layers. The network’s input is 150,528-dimensional, and the number of neurons in the network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264–4096–4096–1000.

一个核映射中的神经元）。第二个卷积层以第一个卷积层的（响应归一化和池化后的）输出作为输入，并用 256 个大小为 5×5×48 的核进行滤波。第三、第四和第五卷积层之间没有插入任何池化或归一化层。第三个卷积层有 384 个大小为 3×3×256 的卷积核，连接到第二个卷积层的（归一化、池化后的）输出。第四个卷积层有 384 个大小为 3×3×192 的卷积核，第五个卷积层有 256 个大小为 3×3×192 的卷积核。全连接层每层有 4096 个神经元。

neurons in a kernel map). The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size \(5 \times 5 \times 48\). The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size \(3 \times 3 \times 256\) connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth convolutional layer has 384 kernels of size 3 × 3 × 192, and the fifth convolutional layer has 256 kernels of size 3 × 3 × 192. The fully-connected layers have 4096 neurons each.

4 减少过拟合 4 Reducing Overfitting

我们的神经网络架构有 6000 万个参数。尽管 ILSVRC 的 1000 个类别使得每个训练样本对图像到标签的映射施加了 10 比特的约束，但事实证明这不足以学习这么多参数而不出现严重的过拟合。下面，我们描述对抗过拟合的两种主要方法。

Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label, this turns out to be insufficient to learn so many parameters without considerable overfitting. Below, we describe the two primary ways in which we combat overfitting.

减少图像数据过拟合最简单且最常用的方法是通过标签保持变换（例如，[25, 4, 5]）人为地扩大数据集。我们采用两种不同形式的数据增强，两者都允许以非常少的计算从原始图像生成变换后的图像，因此变换后的图像不需要存储在磁盘上。在我们的实现中，变换后的图像是在 CPU 上用 Python 代码生成的，同时 GPU 正在训练前一批图像。因此，这些数据增强方案实际上是计算免费的。第一种数据增强形式包括生成图像平移和水平翻转。我们通过从

The easiest and most common method to reduce overfitting on image data is to artificially enlarge the dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two distinct forms of data augmentation, both of which allow transformed images to be produced from the original images with very little computation, so the transformed images do not need to be stored on disk. In our implementation, the transformed images are generated in Python code on the CPU while the GPU is training on the previous batch of images. So these data augmentation schemes are, in effect, computationally free. The first form of data augmentation consists of generating image translations and horizontal reflections. We do this by extracting random 224 × 224 patches (and their horizontal reflections) from the

256×256 图像中提取随机的 224×224 块（及其水平翻转）并在这些提取的块上训练网络来实现这一点。这使我们的训练集大小增加了 2048 倍，尽管生成的训练样本当然是高度相互依赖的。如果没有这种方案，我们的网络会遭受严重的过拟合，这将迫使我们使用更小的网络。在测试时，网络通过提取五个 224×224 块（四个角块和中心块）及其水平翻转（因此总共十个块），并对网络 softmax 层在十个块上的预测进行平均来做出预测。第二种数据增强形式包括改变训练图像中 RGB 通道的强度。具体来说，我们对整个 ImageNet 训练集中的 RGB 像素值集合进行 PCA。对于每张训练图像，我们添加找到的主成分的倍数，

256 × 256 images and training our network on these extracted patches. This increases the size of our training set by a factor of 2048, though the resulting training examples are, of course, highly inter-dependent. Without this scheme, our network suffers from substantial overfitting, which would have forced us to use much smaller networks. At test time, the network makes a prediction by extracting five 224 × 224 patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions made by the network's softmax layer on the ten patches. The second form of data augmentation consists of altering the intensities of the RGB channels in training images. Specifically, we perform PCA on the set of RGB pixel values throughout the ImageNet training set. To each training image, we add multiples of the found principal components,

其幅度与对应特征值乘以一个从均值为零、标准差为 0.1 的高斯分布中抽取的随机变量成正比。因此，对于每个 RGB 图像像素\(I_{xy} = [I_{xy}^R, I_{xy}^G, I_{xy}^B]^T\)，我们添加以下量：

with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from a Gaussian with mean zero and standard deviation 0.1. Therefore to each RGB image pixel \(I_{xy} = [I_{xy}^R, I_{xy}^G, I_{xy}^B]^T\) we add the following quantity:

其中\(p_i\)和\(\lambda_i\)分别是 RGB 像素值的\(3 imes 3\)协方差矩阵的第\(i\)个特征向量和特征值，\(\alpha_i\)是前述随机变量。每个\(\alpha_i\)只对特定训练图像的所有像素抽取一次，直到该图像再次用于训练时重新抽取。该方案大致捕捉了自然图像的一个重要性质，即物体身份对照明强度和颜色的变化具有不变性。该方案将 top-1 错误率降低了超过 1%。

where \(p_i\) and \(\lambda_i\) are the \(i\)-th eigenvector and eigenvalue of the \(3 imes 3\) covariance matrix of RGB pixel values, respectively, and \(\alpha_i\) is the aforementioned random variable. Each \(\alpha_i\) is drawn only once for all the pixels of a particular training image until that image is used for training again, at which point it is re-drawn. This scheme approximately captures an important property of natural images, namely, that object identity is invariant to changes in the intensity and color of the illumination. This scheme reduces the top-1 error rate by over 1%.

结合多个不同模型的预测是一种非常成功的减少测试错误的方法[1, 3]，但对于已经需要数天训练的大型神经网络来说，这似乎成本过高。然而，有一种非常高效的模型组合版本，在训练期间仅增加约两倍的计算成本。最近引入的技术称为“dropout”[10]，其做法是以概率 0.5 将每个隐藏神经元的输出置为零。以这种方式“丢弃”的神经元不参与前向传播和反向传播。因此，每次输入一个样本时，神经网络都会采样一个不同的架构，但这些架构共享权重。这种技术减少了神经元的复杂共适应，因为一个神经元不能依赖特定其他神经元的存在。因此，它被迫学习更鲁棒的特征，这些特征在与许多不同随机子集的神经元结合时都有用。在测试时，我们使用所有神经元，但将其输出乘以 0.5，这是对指数级多个 dropout 网络产生的预测分布取几何平均的一个合理近似。我们在图 2 的前两个全连接层中使用了 dropout。没有 dropout 时，我们的网络表现出严重的过拟合。Dropout 大致将收敛所需的迭代次数增加了一倍。

Combining the predictions of many different models is a very successful way to reduce test errors [1, 3], but it appears to be too expensive for big neural networks that already take several days to train. There is, however, a very efficient version of model combination that only costs about a factor of two during training. The recently-introduced technique, called “dropout” [10], consists of setting to zero the output of each hidden neuron with probability 0.5. The neurons which are “dropped out” in this way do not contribute to the forward pass and do not participate in back-propagation. So every time an input is presented, the neural network samples a different architecture, but all these architectures share weights. This technique reduces complex co-adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons. At test time, we use all the neurons but multiply their outputs by 0.5, which is a reasonable approximation to taking the geometric mean of the predictive distributions produced by the exponentially-many dropout networks. We use dropout in the first two fully-connected layers of Figure 2. Without dropout, our network exhibits substantial overfitting. Dropout roughly doubles the number of iterations required to converge.

图 3：96 个大小为

Figure 3: 96 convolutional kernels of size

5 学习细节 5 Details of learning

我们使用随机梯度下降训练模型，批量大小为 128 个样本，动量为 0.9，权重衰减为 0.0005。我们发现少量的权重衰减对模型学习很重要。换言之，此处的权重衰减不仅仅是正则化器：它减少了模型的训练误差。权重 \(w\) 的更新规则为

We trained our models using stochastic gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005. We found that this small amount of weight decay was important for the model to learn. In other words, weight decay here is not merely a regularizer: it reduces the model's training error. The update rule for weight \(w\) was

其中 \(i\) 是迭代索引，\(v\) 是动量变量，\(\epsilon\) 是学习率，其中 \(\left\langle\frac{\partial L}{\partial w}\big|_{w_i}\right\rangle_{D_i}\) 是目标函数关于 \(w\) 的导数在第 \(i\) 个批次 \(D_i\) 上的均值，在 \(w_i\) 处求值。我们从均值为零、标准差为 0.01 的高斯分布初始化每一层的权重。我们在第二、四、五卷积层以及全连接隐藏层中将神经元偏置初始化为常数 1。这种初始化通过为 ReLU 提供正输入加速了学习的早期阶段。我们将剩余层的神经元偏置初始化为常数 0。我们对所有层使用相同的学习率，并在训练过程中手动调整。遵循的启发式方法是：当验证错误率在当前学习率下停止改善时，将学习率除以 10。学习率初始化为 0.01，并在终止前降低三次。我们训练网络大约 90 个周期，遍历 120 万张图像的训练集，在两张 NVIDIA GTX 580 3GB GPU 上耗时五到六天。

where \(i\) is the iteration index, \(v\) is the momentum variable, \(\epsilon\) is the learning rate, and \(\left\langle\frac{\partial L}{\partial w}\big|_{w_i}\right\rangle_{D_i}\) is the average over the \(i\)th batch \(D_i\) of the derivative of the objective with respect to \(w\), evaluated at \(w_i\). We initialized the weights in each layer from a zero-mean Gaussian distribution with standard deviation 0.01. We initialized the neuron biases in the second, fourth, and fifth convolutional layers, as well as in the fully-connected hidden layers, with the constant 1. This initialization accelerates the early stages of learning by providing the ReLUs with positive inputs. We initialized the neuron biases in the remaining layers with the constant 0. We used an equal learning rate for all layers, which we adjusted manually throughout training. The heuristic which we followed was to divide the learning rate by 10 when the validation error rate stopped improving with the current learning rate. The learning rate was initialized at 0.01 and reduced three times prior to termination. We trained the network for roughly 90 cycles through the training set of 1.2 million images, which took five to six days on two NVIDIA GTX 580 3GB GPUs.

6 结果 6 Results

我们在 ILSVRC-2010 上的结果总结在表 1 中。我们的网络在测试集上达到了 37.5% 的 top-1 错误率和 17.0% 的 top-5 错误率。ILSVRC-2010 竞赛中的最佳性能是 47.1% 和 28.2%，该方法对从不同特征上训练的六个稀疏编码模型产生的预测进行平均 [2]。此后，最佳已发布结果为 45.7% 和 25.7%，该方法对从两类密集采样特征计算的 Fisher 向量 (FV) 上训练的两个分类器的预测进行平均 [24]。

Our results on ILSVRC-2010 are summarized in Table 1. Our network achieves top-1 and top-5 test set error rates of 37.5% and 17.0%. The best performance achieved during the ILSVRC-2010 competition was 47.1% and 28.2% with an approach that averages the predictions produced from six sparse-coding models trained on different features [2], and since then the best published results are 45.7% and 25.7% with an approach that averages the predictions of two classifiers trained on Fisher Vectors (FVs) computed from two types of densely-sampled features [24].

稀疏编码 [2] 47.1% 28.2% SIFT + FVs [24] 45.7% 25.7%

Sparse coding [2] 47.1% 28.2% SIFT + FVs [24] 45.7% 25.7%

表 1: ILSVRC-2010 测试集结果比较。斜体为其他方法取得的最佳结果。

Table 1: Comparison of results on ILSVRC-2010 test set. In italics are best results achieved by others.

我们还将模型提交到 ILSVRC-2012 竞赛，并在表 2 中报告结果。由于 ILSVRC-2012 测试集标签不公开，我们无法报告尝试的所有模型的测试错误率。在本段剩余部分，我们互换使用验证和测试错误率，因为根据经验它们相差不超过 0.1% (见表 2)。本文描述的 CNN 达到了 18.2% 的 top-5 错误率。对五个类似 CNN 的预测进行平均得到 16.4% 的错误率。训练一个 CNN，在最后一个池化层之上增加第六个卷积层，对整个 ImageNet Fall 2011 发布版 (1500 万图像，22000 类别) 进行分类，然后在 ILSVRC-2012 上进行“微调”，得到 16.6% 的错误率。将两个在 Fall 2011 发布版上预训练的 CNN 与上述五个 CNN 的预测进行平均，得到 15.3% 的错误率。竞赛第二名以 26.2% 的错误率，通过对从不同类型的密集采样特征计算的 FV 上训练的多个分类器的预测进行平均 [7]。

We also entered our model in the ILSVRC-2012 competition and report our results in Table 2. Since the ILSVRC-2012 test set labels are not publicly available, we cannot report test error rates for all the models that we tried. In the remainder of this paragraph, we use validation and test error rates interchangeably because in our experience they do not differ by more than 0.1% (see Table 2). The CNN described in this paper achieves a top-5 error rate of 18.2%. Averaging the predictions of five similar CNNs gives an error rate of 16.4%. Training one CNN, with an extra sixth convolutional layer over the last pooling layer, to classify the entire ImageNet Fall 2011 release (15M images, 22K categories), and then “fine-tuning” it on ILSVRC-2012 gives an error rate of 16.6%. Averaging the predictions of two CNNs that were pre-trained on the entire Fall 2011 release with the aforementioned five CNNs gives an error rate of 15.3%. The second-best contest entry achieved an error rate of 26.2% with an approach that averages the predictions of several classifiers trained on FVs computed from different types of densely-sampled features [7].

模型 Top-1 (val) Top-5 (val) Top-5 (test)

Model Top-1 (val) Top-5 (val) Top-5 (test)

1 个 CNN 40.7% 18.2% — 5 个 CNN 38.1% 16.4% 16.4%

1 CNN 40.7% 18.2% — 5 CNNs 38.1% 16.4% 16.4%

1 个 CNN* 39.0% 16.6% — 7 个 CNN* 36.7% 15.4% 15.3%

1 CNN* 39.0% 16.6% — 7 CNNs* 36.7% 15.4% 15.3%

表 2：在 ILSVRC-2012 验证集和测试集上的错误率比较。斜体为其他研究者取得的最佳结果。带星号*的模型是“预训练”的，用于对整个 ImageNet 2011 秋季发布版本进行分类。详见第 6 节。

Table 2: Comparison of error rates on ILSVRC-2012 validation and test sets. In italics are best results achieved by others. Models with an asterisk* were “pre-trained” to classify the entire ImageNet 2011 Fall release. See Section 6 for details.

最后，我们还在 ImageNet 2009 秋季版本上报告了错误率，该版本有 10,184 个类别和 890 万张图像。在这个数据集上，我们遵循文献惯例，使用一半图像进行训练，一半进行测试。由于没有固定的测试集，我们的划分方式必然与先前作者使用的划分不同，但这不会显著影响结果。在此数据集上，我们的 top-1 和 top-5 错误率为 67.4%和

Finally, we also report our error rates on the Fall 2009 version of ImageNet with 10,184 categories and 8.9 million images. On this dataset we follow the convention in the literature of using half of the images for training and half for testing. Since there is no established test set, our split necessarily differs from the splits used by previous authors, but this does not affect the results appreciably. Our top-1 and top-5 error rates on this dataset are 67.4% and

40.9%，由上述网络实现，但在最后一个池化层之上增加了一个额外的第六个卷积层。该数据集上此前发布的最佳结果为 78.1%和 60.9% [19]。

40.9%, attained by the net described above but with an additional, sixth convolutional layer over the last pooling layer. The best published results on this dataset are 78.1% and 60.9% [19].

图 3 展示了网络两个数据连接层学到的卷积核。网络学到了各种频率和方向选择性核，以及各种彩色斑块。注意两个 GPU 展现出的特化现象，这是第 3.5 节所述受限连接的结果。GPU 1 上的核基本上与颜色无关，而 GPU 2 上的核则主要针对特定颜色。这种特化在每次运行中都会发生，并且与任何特定的随机权重初始化无关（仅 GPU 编号可能不同）。

Figure 3 shows the convolutional kernels learned by the network’s two data-connected layers. The network has learned a variety of frequency- and orientation-selective kernels, as well as various colored blobs. Notice the specialization exhibited by the two GPUs, a result of the restricted connectivity described in Section 3.5. The kernels on GPU 1 are largely color-agnostic, while the kernels on GPU 2 are largely color-specific. This kind of specialization occurs during every run and is independent of any particular random weight initialization (modulo a renumbering of the GPUs).

图 4：（左）八张 ILSVRC-2010 测试图像以及模型认为最可能的五个标签。每张图像下方标注了正确标签，正确标签被赋予的概率也用红色条显示（如果它恰好在 top-5 中）。（右）第一列为五张 ILSVRC-2010 测试图像。其余列展示了六个训练图像，这些训练图像在最后一个隐藏层产生的特征向量与测试图像的特征向量之间的欧氏距离最小。

Figure 4: (Left) Eight ILSVRC-2010 test images and the five labels considered most probable by our model. The correct label is written under each image, and the probability assigned to the correct label is also shown with a red bar (if it happens to be in the top 5). (Right) Five ILSVRC-2010 test images in the first column. The remaining columns show the six training images that produce feature vectors in the last hidden layer with the smallest Euclidean distance from the feature vector for the test image.

在图 4 的左图中，我们通过计算模型在八张测试图像上的 top-5 预测，定性评估了网络所学到的内容。注意，即使偏离中心的物体（如左上角的螨虫）也能被网络识别。大多数 top-5 标签看起来合理。例如，对于豹子，仅其他类型的猫被认为可能是合理的标签。在某些情况下（格栅、樱桃），照片的预期焦点确实存在歧义。另一种探究网络视觉知识的方法是考虑图像在最后一个 4096 维隐藏层引起的特征激活。如果两张图像产生的特征激活向量具有较小的欧氏距离，我们可以说神经网络的高层认为它们相似。图 4 展示了测试集中的五张图像以及根据此度量与每张图像最相似的六张训练图像。注意，在像素层面，检索到的训练图像通常与第一列中的查询图像在 L2 距离上不接近。例如，检索到的狗和大象出现了多种姿态。我们在补充材料中展示了更多测试图像的结果。使用两个 4096 维实值向量之间的欧氏距离来计算相似度是低效的，但可以通过训练自编码器将这些向量压缩为短二进制码来提高效率。这应该会产生比将自编码器应用于原始像素[14]更好的图像检索方法，因为原始像素方法未利用图像标签，因此倾向于检索具有相似边缘图案的图像，而不管它们是否语义相似。

In the left panel of Figure 4 we qualitatively assess what the network has learned by computing its top-5 predictions on eight test images. Notice that even off-center objects, such as the mite in the top-left, can be recognized by the net. Most of the top-5 labels appear reasonable. For example, only other types of cat are considered plausible labels for the leopard. In some cases (grille, cherry) there is genuine ambiguity about the intended focus of the photograph. Another way to probe the network’s visual knowledge is to consider the feature activations induced by an image at the last, 4096-dimensional hidden layer. If two images produce feature activation vectors with a small Euclidean separation, we can say that the higher levels of the neural network consider them to be similar. Figure 4 shows five images from the test set and the six images from the training set that are most similar to each of them according to this measure. Notice that at the pixel level, the retrieved training images are generally not close in L2 to the query images in the first column. For example, the retrieved dogs and elephants appear in a variety of poses. We present the results for many more test images in the supplementary material. Computing similarity by using Euclidean distance between two 4096-dimensional, real-valued vectors is inefficient, but it could be made efficient by training an auto-encoder to compress these vectors to short binary codes. This should produce a much better image retrieval method than applying auto-encoders to the raw pixels [14], which does not make use of image labels and hence has a tendency to retrieve images with similar patterns of edges, whether or not they are semantically similar.

讨论 7 Discussion

我们的结果表明，一个大型深度卷积神经网络能够在极具挑战性的数据集上，仅通过监督学习实现创纪录的结果。值得注意的是，如果移除任何一个卷积层，网络的性能都会下降。例如，移除任何中间层会导致网络 top-1 性能下降约 2%。因此，深度对于实现我们的结果确实很重要。为了简化实验，我们没有使用任何无监督预训练，尽管我们预计它会有所帮助，特别是在我们获得足够的算力以显著增加网络规模，而无需相应增加标注数据量的情况下。到目前为止，随着我们使网络更大并训练更长时间，我们的结果不断改善，但要匹配人类视觉系统的颞下通路，我们仍需多个数量级的提升。最终，我们希望将非常大且深的卷积网络应用于视频序列，其中时间结构提供了静态图像中缺失或远不那么明显的有用信息。

Our results show that a large, deep convolutional neural network is capable of achieving record-breaking results on a highly challenging dataset using purely supervised learning. It is notable that our network's performance degrades if a single convolutional layer is removed. For example, removing any of the middle layers results in a loss of about 2% for the top-1 performance of the network. So the depth really is important for achieving our results. To simplify our experiments, we did not use any unsupervised pre-training even though we expect that it will help, especially if we obtain enough computational power to significantly increase the size of the network without obtaining a corresponding increase in the amount of labeled data. Thus far, our results have improved as we have made our network larger and trained it longer but we still have many orders of magnitude to go in order to match the infero-temporal pathway of the human visual system. Ultimately we would like to use very large and deep convolutional nets on video sequences where the temporal structure provides very helpful information that is missing or far less obvious in static images.

ESANN, 2011. [15] Y. Le Cun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel, 等. Handwritten digit recognition with a back-propagation network. 载于 Advances in neural information processing systems, 1990. [16] Y. LeCun, F.J. Huang, 和 L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. 载于 Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, 卷 2, 页码 II–97. IEEE, 2004. [17] Y. LeCun, K. Kavukcuoglu, 和 C. Farabet. Convolutional networks and applications in vision. 载于

ESANN, 2011. [15] Y. Le Cun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel, et al. Handwritten digit recognition with a back-propagation network. In Advances in neural information processing systems, 1990. [16] Y. LeCun, F.J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, volume 2, pages II–97. IEEE, 2004. [17] Y. LeCun, K. Kavukcuoglu, and C. Farabet. Convolutional networks and applications in vision. In

ESANN, 2011. [15] Y. Le Cun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel, 等. Handwritten digit recognition with a back-propagation network. 载于 Advances in neural information processing systems, 1990. [16] Y. LeCun, F.J. Huang, 和 L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. 载于 Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, 卷 2, 页码 II–97. IEEE, 2004. [17] Y. LeCun, K. Kavukcuoglu, 和 C. Farabet. Convolutional networks and applications in vision. 载于

ESANN, 2011. [15] Y. Le Cun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel, et al. Handwritten digit recognition with a back-propagation network. In Advances in neural information processing systems, 1990. [16] Y. LeCun, F.J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, volume 2, pages II–97. IEEE, 2004. [17] Y. LeCun, K. Kavukcuoglu, and C. Farabet. Convolutional networks and applications in vision. In

Our results show that a large, deep convolutional neural network is capable of achieving record-breaking results on a highly challenging dataset using purely supervised learning. It is notable that our network’s performance degrades if a single convolutional layer is removed. For example, removing any of the middle layers results in a loss of about 2% for the top-1 performance of the network. So the depth really is important for achieving our results. To simplify our experiments, we did not use any unsupervised pre-training even though we expect that it will help, especially if we obtain enough computational power to significantly increase the size of the network without obtaining a corresponding increase in the amount of labeled data. Thus far, our results have improved as we have made our network larger and trained it longer but we still have many orders of magnitude to go in order to match the infero-temporal pathway of the human visual system. Ultimately we would like to use very large and deep convolutional nets on video sequences where the temporal structure provides very helpful information that is missing or far less obvious in static images. 8References

电路与系统国际研讨会（ISCAS），2010 年 IEEE 国际研讨会论文集，页码 253–256。IEEE，2010 年。[18] H. Lee, R. Grosse, R. Ranganath, 和 A.Y. Ng。用于分层表示的可扩展无监督学习的卷积深度信念网络。第 26 届国际机器学习年会论文集，页码 609–616。ACM，2009 年。[19] T. Mensink, J. Verbeek, F. Perronnin, 和 G. Csurka。大规模图像分类的度量学习：以近乎零成本泛化到新类别。欧洲计算机视觉会议（ECCV），意大利佛罗伦萨，2012 年 10 月。[20] V. Nair 和 G. E. Hinton。修正线性单元改进受限玻尔兹曼机。第 27 届国际机器学习会议论文集，2010 年。[21] N. Pinto, D.D. Cox, 和 J.J. DiCarlo。为什么真实世界的视觉物体识别很难？PLoS 计算生物学，4(1):e27，2008 年。[22] N. Pinto, D. Doukhan, J.J. DiCarlo, 和 D.D. Cox。一种高通量筛选方法，用于发现良好的生物启发视觉表示形式。PLoS 计算生物学，5(11):e1000579，2009 年。[23] B.C. Russell, A. Torralba, K.P. Murphy, 和 W.T. Freeman。Labelme：一种用于图像标注的数据库和基于 Web 的工具。国际计算机视觉杂志，77(1):157–173，2008 年。[24] J. Sánchez 和 F. Perronnin。大规模图像分类的高维签名压缩。2011 年 IEEE 计算机视觉与模式识别会议（CVPR），页码 1665–1672。IEEE，2011 年。[25] P.Y. Simard, D. Steinkraus, 和 J.C. Platt。应用于视觉文档分析的卷积神经网络最佳实践。第七届国际文档分析与识别会议论文集，第 2 卷，页码 958–962，2003 年。[26] S.C. Turaga, J.F. Murray, V. Jain, F. Roth, M. Helmstaedter, K. Briggman, W. Denk, 和 H.S. Seung。卷积网络可以学习生成用于图像分割的亲和力图。神经计算，22(2):511–538，2010 年。

Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium on, pages 253–256. IEEE, 2010. [18] H. Lee, R. Grosse, R. Ranganath, and A.Y. Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 609–616. ACM, 2009. [19] T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka. Metric Learning for Large Scale Image Classification: Generalizing to New Classes at Near-Zero Cost. In ECCV - European Conference on Computer Vision, Florence, Italy, October 2012. [20] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proc. 27th International Conference on Machine Learning, 2010. [21] N. Pinto, D.D. Cox, and J.J. DiCarlo. Why is real-world visual object recognition hard? PLoS computational biology, 4(1):e27, 2008. [22] N. Pinto, D. Doukhan, J.J. DiCarlo, and D.D. Cox. A high-throughput screening approach to discovering good forms of biologically inspired visual representation. PLoS computational biology, 5(11):e1000579, 2009. [23] B.C. Russell, A. Torralba, K.P. Murphy, and W.T. Freeman. Labelme: a database and web-based tool for image annotation. International journal of computer vision, 77(1):157–173, 2008. [24] J. Sánchez and F. Perronnin. High-dimensional signature compression for large-scale image classification. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1665–1672. IEEE, 2011. [25] P.Y. Simard, D. Steinkraus, and J.C. Platt. Best practices for convolutional neural networks applied to visual document analysis. In Proceedings of the Seventh International Conference on Document Analysis and Recognition, volume 2, pages 958–962, 2003. [26] S.C. Turaga, J.F. Murray, V. Jain, F. Roth, M. Helmstaedter, K. Briggman, W. Denk, and H.S. Seung. Convolutional networks can learn to generate affinity graphs for image segmentation. Neural Computation, 22(2):511–538, 2010.