我们介绍了 Adam，一种基于低阶矩自适应估计的随机目标函数一阶梯度优化算法。该方法易于实现，计算效率高，内存需求小，对梯度的对角缩放具有不变性，特别适合处理数据和/或参数规模较大的问题。该方法也适用于非平稳目标以及噪声和/或梯度稀疏的问题。超参数具有直观的解释，通常只需很少的调整。我们还讨论了启发 Adam 的相关算法的一些联系。此外，我们分析了该算法的理论收敛性质，并给出了在线凸优化框架下与已知最优结果相当的收敛率遗憾界。实验结果表明，Adam 在实际应用中表现良好，与其他随机优化方法相比具有优势。最后，我们讨论了 AdaMax，一种基于无穷范数的 Adam 变体。
We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.
核心贡献 · Key contributions
提出 Adam 算法，结合 AdaGrad 和 RMSProp，实现自适应学习率。 Proposes Adam, combining AdaGrad and RMSProp for adaptive learning rates.
引入矩估计的偏差校正，防止初始大步长。 Introduces bias correction for moment estimates to prevent large initial steps.
提供在线凸优化下的后悔界，保证收敛。 Provides regret bound in online convex optimization, ensuring convergence.
实验证明在逻辑回归、多层感知机和 CNN 上表现优异。 Empirically shows superior performance on logistic regression, MLPs, and CNNs.