Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at https://github.com/loshchil/SGDR
核心贡献 · Key contributions
提出 SGDR，一种用于 SGD 的热重启技术，以提升深度神经网络训练中的实时性能。 Proposes SGDR, a warm restart technique for SGD to improve anytime performance in training deep neural networks.
在 CIFAR-10（3.14%）和 CIFAR-100（16.21%）上使用快照集成取得了新的最优结果。 Achieves new state-of-the-art results on CIFAR-10 (3.14%) and CIFAR-100 (16.21%) using ensembles of snapshots.
在 CIFAR 数据集上，相比标准学习率调度，收敛速度提升 2 到 4 倍。 Demonstrates 2x to 4x faster convergence compared to standard learning rate schedules on CIFAR datasets.
在脑电图记录和下采样 ImageNet 上验证 SGDR，展示了更好的实时性能。 Validates SGDR on EEG recordings and downsampled ImageNet, showing better anytime performance.
表明 SGDR 能够从中间模型快照中高效构建集成，无需额外成本。 Shows that SGDR enables efficient ensemble building from intermediate model snapshots at no extra cost.
提供开源代码以确保可复现性。 Provides open-source code for reproducibility.
局限 · Limitations
实证研究仅限于 CIFAR-10、CIFAR-100、脑电图和下采样 ImageNet 数据集。 Empirical study limited to CIFAR-10, CIFAR-100, EEG, and downsampled ImageNet datasets.
缺乏对 SGDR 收敛性或泛化保证的理论分析。 No theoretical analysis of convergence or generalization guarantees for SGDR.
初始学习率和重启周期的超参数敏感性未充分探索。 Hyperparameter sensitivity of initial learning rate and restart period not fully explored.
结果可能无法推广到其他架构或除带动量 SGD 之外的优化方法。 Results may not generalize to other architectures or optimization methods beyond SGD with momentum.
集成性能依赖于多次运行和快照选择，增加了计算成本。 Ensemble performance relies on multiple runs and snapshot selection, increasing computational cost.
论文章节 · Sections（共 13）
摘要Abstract
1 引言1 Introduction
2.1 无梯度优化中的重启2.1 Restarts in gradient-free optimization
2.2 基于梯度的优化中的重启2.2 Restarts in gradient-based optimization
3 带热重启的随机梯度下降（SGDR）3 Stochastic Gradient Descent with warm restarts (SGDR)
4.1 实验设置4.1 Experimental settings
4.2 单模型结果4.2 Single-Model Results
4.3 集成结果4.3 Ensemble Results
4.4 脑电图记录数据集上的实验4.4 Experiments on a dataset of EEG recordings
4.5 下采样 ImageNet 数据集上的初步实验4.5 Preliminary experiments on a downsampled ImageNet dataset