L2 正则化与权重衰减正则化在标准随机梯度下降中（按学习率重新缩放后）是等价的，但我们证明对于自适应梯度算法（如 Adam）并非如此。尽管这些算法的常见实现采用 L2 正则化（由于我们揭示的不等价性，将其称为“权重衰减”可能具有误导性），我们提出了一种简单的修改，通过将权重衰减与相对于损失函数的优化步骤解耦，恢复权重衰减正则化的原始形式。我们提供的经验证据表明，我们提出的修改（i）将权重衰减因子的最优选择与标准 SGD 和 Adam 的学习率设置解耦，（ii）显著提高了 Adam 的泛化性能，使其能够与带动量的 SGD 在图像分类数据集上竞争（此前 Adam 通常被后者超越）。我们提出的解耦权重衰减已被许多研究者采用，社区已在 TensorFlow 和 PyTorch 中实现；我们实验的完整源代码可在 https://github.com/loshchil/AdamW-and-SGDW 获取。
L$_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is \emph{not} the case for adaptive gradient algorithms, such as Adam. While common implementations of these algorithms employ L$_2$ regularization (often calling it "weight decay" in what may be misleading due to the inequivalence we expose), we propose a simple modification to recover the original formulation of weight decay regularization by \emph{decoupling} the weight decay from the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). Our proposed decoupled weight decay has already been adopted by many researchers, and the community has implemented it in TensorFlow and PyTorch; the complete source code for our experiments is available at https://github.com/loshchil/AdamW-and-SGDW
核心贡献 · Key contributions
发现 L2 正则化与权重衰减在自适应梯度方法（如 Adam）中不等价。 Identified inequivalence of L2 regularization and weight decay for adaptive gradient methods like Adam.
提出解耦权重衰减（AdamW）以恢复原始权重衰减形式。 Proposed decoupled weight decay (AdamW) to recover original weight decay formulation.
证明解耦权重衰减提升 Adam 泛化性能，在图像分类上媲美带动量的 SGD。 Showed decoupled weight decay improves Adam's generalization, matching SGD with momentum on image classification.
展示解耦权重衰减使学习率和权重衰减超参数更独立。 Demonstrated decoupled weight decay makes learning rate and weight decay hyperparameters more independent.
在 CIFAR-10 和 ImageNet32x32 上使用多种调度和预算进行实证验证。 Empirically validated on CIFAR-10 and ImageNet32x32 with various schedules and budgets.
结合热重启（AdamWR）以提升任意时刻性能。 Extended with warm restarts (AdamWR) for improved anytime performance.
局限 · Limitations
结果仅在图像分类数据集上验证，需在更广泛任务上验证。 Results verified only on image classification datasets; need broader task validation.
理论分析限于简单固定预条件器情况。 Theoretical analysis limited to a simple fixed preconditioner case.
实证聚焦 Adam，未直接测试 AdaGrad 等其他自适应方法。 Empirical focus on Adam; other adaptive methods like AdaGrad not directly tested.
最优权重衰减依赖于总批次数，需调参。 Optimal weight decay depends on total batch passes, requiring tuning.
改进可能不适用于正则化不重要的任务。 Improvement may not hold for tasks where regularization is less important.
论文章节 · Sections（共 12）
摘要Abstract
1 引言1 Introduction
2 将权重衰减与基于梯度的更新解耦2 Decoupling the Weight Decay from the Gradient-based Update
3 通过自适应梯度方法的贝叶斯滤波视角证明解耦权重衰减的合理性3 Justification of Decoupled Weight Decay via a View of Adaptive Gradient Methods as Bayesian Filtering
4 实验验证4 Experimental Validation
4.1 评估不同学习率调度下的解耦权重衰减4.1 Evaluating Decoupled Weight Decay With Different Learning Rate Schedules
4.2 解耦权重衰减与初始学习率参数4.2 Decoupling the Weight Decay and Initial Learning Rate Parameters
4.3 AdamW 的更好泛化性4.3 Better Generalization of AdamW
4.4 带热重启的 AdamWR 以获得更好的随时性能4.4 AdamWR with Warm Restarts for Better Anytime Performance
4.5 AdamW 在其他数据集和架构上的使用4.5 Use of AdamW on other datasets and architectures