在最近提出的几种随机优化方法（如 RMSProp、Adam、Adadelta）中，参数更新通过缩放过去梯度平方的指数移动平均的平方根倒数来实现。维护这些逐参数二阶矩估计需要与参数数量相等的内存。对于神经网络权重矩阵的情况，我们提出仅维护这些移动平均的行和与列和，并基于这些和估计逐参数二阶矩。我们通过实验证明，该方法产生与基线相似的结果。其次，我们表明当二阶矩累积器的衰减率过慢时，自适应方法可能产生过大的更新。我们提出更新裁剪和逐渐增加衰减率的方案作为补救措施。结合这些方法并去除动量，我们在训练 WMT 2014 英德机器翻译任务的 Transformer 模型时，取得了与已发表的 Adam 方案相当的结果，同时优化器使用的辅助存储非常少。最后，我们提出根据参数本身的尺度来缩放参数更新。
In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-row and per-column sums of these moving averages, and estimating the per-parameter second moments based on these sums. We demonstrate empirically that this method produces similar results to the baseline. Secondly, we show that adaptive methods can produce larger-than-desired updates when the decay rate of the second moment accumulator is too slow. We propose update clipping and a gradually increasing decay rate scheme as remedies. Combining these methods and dropping momentum, we achieve comparable results to the published Adam regime in training the Transformer model on the WMT 2014 English-German machine translation task, while using very little auxiliary storage in the optimizer. Finally, we propose scaling the parameter updates based on the scale of the parameters themselves.
核心贡献 · Key contributions
提出 Adafactor，一种通过分解二阶矩估计实现内存高效的适应性优化器。 Proposes Adafactor, a memory-efficient adaptive optimizer using factored second-moment estimation.
通过存储行和列的和，将矩阵参数的内存从 O(nm)降低到 O(n+m)。 Reduces memory from O(nm) to O(n+m) for matrix parameters by storing row and column sums.
识别出二阶矩衰减过慢导致的不稳定性，并提出更新裁剪作为补救。 Identifies instability from slow second-moment decay and proposes update clipping.
引入逐渐增加的衰减率调度，无需预热即可稳定训练。 Introduces gradually increasing decay rate schedule to stabilize training without warmup.
提出基于参数尺度的相对步长缩放，增强鲁棒性。 Proposes relative step size scaling based on parameter scale for robustness.
在 WMT14 英德翻译任务上达到与 Adam 相当的 BLEU 分数，且内存为次线性。 Achieves comparable BLEU to Adam on WMT14 En-De translation with sublinear memory.
局限 · Limitations
实证验证仅限于机器翻译中的 Transformer 模型，泛化性未证实。 Empirical validation limited to Transformer on machine translation; generalizability unproven.
秩 1 分解可能丢失具有复杂梯度结构的矩阵的信息。 Rank-1 factorization may lose information for matrices with complex gradient structure.
更新裁剪阈值 d 需要调参，固定值可能不适用于所有任务。 Update clipping threshold d requires tuning; fixed value may not suit all tasks.
相对步长假设参数尺度有意义，可能对零初始化参数失效。 Relative step size assumes parameter scale is meaningful; may fail for zero-initialized parameters.
未提供非凸设置下的理论收敛保证。 Theoretical convergence guarantees not provided for non-convex settings.
论文章节 · Sections（共 22）
摘要Abstract
1 引言与背景1 Introduction and Background
2 Adam 简要回顾2 A Brief Review of Adam
3 分解二阶矩估计3 Factored Second Moment Estimation
3.1 与先前工作的关系3.1 Relation to Prior Work
3.2 实验3.2 Experiments
4 无动量4 No Momentum
4.1 实验4.1 Experiments
5 Adam 的问题：过时的二阶矩估计5 A Problem with Adam: Out-of-Date Second Moment Estimator