学习率预热启发式方法在稳定训练、加速收敛以及提升 RMSprop 和 Adam 等自适应随机优化算法的泛化能力方面取得了显著成功。本文详细研究了其机制。在探索预热背后的理论时，我们发现了自适应学习率的一个问题（即在早期阶段其方差过大），提出预热作为一种方差缩减技术，并通过实验和理论证据验证了我们的假设。我们进一步提出了 RAdam，这是 Adam 的一个新变体，通过引入一项来修正自适应学习率的方差。在图像分类、语言建模和神经机器翻译上的大量实验结果验证了我们的直觉，并展示了所提方法的有效性和鲁棒性。所有实现均可从 https://github.com/LiyuanLucasLiu/RAdam 获取。
The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Here, we study its mechanism in details. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate (i.e., it has problematically large variance in the early stage), suggest warmup works as a variance reduction technique, and provide both empirical and theoretical evidence to verify our hypothesis. We further propose RAdam, a new variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate. Extensive experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the effectiveness and robustness of our proposed method. All implementations are available at: https://github.com/LiyuanLucasLiu/RAdam.
核心贡献 · Key contributions
发现自适应学习率在训练早期方差过大是收敛问题的根本原因。 Identifies large variance of adaptive learning rate in early training as root cause of convergence issues.
为预热启发式方法作为方差缩减技术提供了理论依据。 Provides theoretical justification for warmup heuristic as variance reduction technique.
提出 RAdam，一种通过修正自适应学习率方差的新 Adam 变体。 Proposes RAdam, a new Adam variant that rectifies adaptive learning rate variance.
在图像分类、语言建模和神经机器翻译上验证了 RAdam 的有效性和鲁棒性。 Demonstrates RAdam's effectiveness and robustness across image classification, language modeling, and NMT.
表明 RAdam 降低了对学习率和预热步长超参数的敏感性。 Shows RAdam reduces sensitivity to learning rate and warmup length hyperparameters.
局限 · Limitations
分析假设梯度为独立同分布零均值高斯，实际中可能不成立。 Analysis assumes gradients are i.i.d. Gaussian with zero mean, which may not hold in practice.
用简单移动平均近似指数移动平均可能引入误差。 Approximation of exponential moving average by simple moving average may introduce error.
方差的一阶近似在样本量极小时可能不准确。 First-order approximation of variance may be inaccurate for very small sample sizes.
RAdam 的修正项依赖超参数β2，需调参以获得最佳性能。 RAdam's rectification term relies on hyperparameter β2, requiring tuning for optimal performance.
论文章节 · Sections（共 16）
摘要Abstract
1 引言1 Introduction
2 预备知识与动机2 Preliminaries and Motivations
3 自适应学习率的方差3 Variance of the Adaptive Learning Rate
3.1 预热作为方差缩减3.1 Warmup as Variance Reduction
3.2 自适应学习率方差分析3.2 Analysis of Adaptive Learning Rate Variance
4 修正的自适应学习率4 Rectified Adaptive Learning Rate
4.1 ρ的估计4.1 Estimation of ρ𝜌\rho
4.2 方差估计与修正4.2 Variance Estimation and Rectification
4.3 与预热及其他稳定技术的比较4.3 In Comparison with Warmup and Other Stabilization Techniques