A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.
核心贡献 · Key contributions
提出知识蒸馏技术，通过带温度缩放的软目标将集成模型的知识迁移到单个小型模型中。 Introduces knowledge distillation, transferring ensemble knowledge into a single small model using soft targets with temperature scaling.
在 MNIST 上，蒸馏模型仅 74 个错误，而正则化小网络为 146 个错误，即使迁移集中没有数字 3 的样本。 Shows distilled model on MNIST achieves 74 errors vs 146 for a regularized small net, even without seeing digit 3 in transfer set.
在商业语音识别中，蒸馏模型达到集成模型的性能，词错误率 10.9%。 Demonstrates distillation improves commercial speech recognition: distilled single model matches ensemble performance with 10.9% WER.
提出专家集成：一个通用模型加多个细粒度类别专家模型，可并行训练，在 JFT 上准确率提升 4.4%。 Proposes specialist ensemble: one generalist plus many specialists for fine-grained classes, trained in parallel, improving JFT accuracy by 4.4%.
软目标防止过拟合：仅 3%语音数据时帧准确率 57%，而硬目标仅 44.5%。 Shows soft targets prevent overfitting with limited data: 3% of speech data yields 57% frame accuracy vs 44.5% with hard targets.