最近提出的几种随机优化方法，如 RMSProp、Adam、Adadelta、Nadam，在训练深度网络时取得了成功，它们都基于使用过去梯度平方的指数移动平均的平方根来缩放梯度更新。在许多应用中，例如具有大输出空间的学习，经验观察发现这些算法无法收敛到最优解（或在非凸设置中无法收敛到临界点）。我们表明，这种失败的一个原因在于算法中使用的指数移动平均。我们提供了一个简单的凸优化设置中的显式例子，其中 Adam 不收敛到最优解，并描述了先前 Adam 算法分析中的确切问题。我们的分析表明，通过赋予这些算法过去梯度的“长期记忆”可以解决收敛问题，并提出了 Adam 算法的新变体，这些变体不仅解决了收敛问题，而且通常还能提高经验性能。
Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings). We show that one cause for such failures is the exponential moving average used in the algorithms. We provide an explicit example of a simple convex optimization setting where Adam does not converge to the optimal solution, and describe the precise problems with the previous analysis of Adam algorithm. Our analysis suggests that the convergence issues can be fixed by endowing such algorithms with `long-term memory' of past gradients, and propose new variants of the Adam algorithm which not only fix the convergence issues but often also lead to improved empirical performance.
核心贡献 · Key contributions
指出 Adam 和 RMSprop 中的指数移动平均导致凸优化中不收敛。 Identifies that exponential moving average in Adam and RMSprop causes non-convergence in convex optimization.
提供显式凸示例，Adam 无法收敛到最优解。 Provides explicit convex example where Adam fails to converge to optimal solution.
提出 AMSGrad 算法，具有过去梯度的长期记忆，修复收敛问题。 Proposes AMSGrad algorithm with long-term memory of past gradients to fix convergence issues.
证明 AMSGrad 在凸设置中的遗憾界，类似于 Adagrad。 Proves regret bound for AMSGrad in convex setting, similar to Adagrad.
表明任何常数β1、β2 满足β1 < √β2 时 Adam 不收敛。 Shows that any constant β1, β2 with β1 < √β2 leads to non-convergence for Adam.
实验证明 AMSGrad 在逻辑回归和神经网络上优于 Adam。 Empirically demonstrates AMSGrad outperforms Adam on logistic regression and neural networks.
局限 · Limitations
不收敛示例是精心构造的，可能不反映典型实际场景。 Non-convergence examples are carefully constructed and may not reflect typical practical scenarios.
理论分析聚焦凸设置；非凸收敛未保证。 Theoretical analysis focuses on convex settings; non-convex convergence is not guaranteed.
AMSGrad 需要调整超参数如β1 和β2，与 Adam 类似。 AMSGrad requires tuning of hyperparameters like β1 and β2, similar to Adam.
实验评估限于 MNIST 和 CIFAR-10；需要更广泛的基准。 Empirical evaluation is limited to MNIST and CIFAR-10; broader benchmarks are needed.
提出的修复因存储过去平方梯度的最大值而略微增加内存。 Proposed fixes increase memory slightly due to storing maximum of past squared gradients.
论文章节 · Sections（共 8）
摘要Abstract
1 引言1 Introduction
2 预备知识2 Preliminaries
3 Adam 的非收敛性3 The Non-Convergence of Adam
4 一种新的指数移动平均变体：AMSGrad4 A New Exponential Moving Average Variant: AMSGrad