A common way to speed up training of large convolutional networks is to add computational units. Training is then performed using data-parallel synchronous Stochastic Gradient Descent (SGD) with mini-batch divided between computational units. With an increase in the number of nodes, the batch size grows. But training with large batch size often results in the lower model accuracy. We argue that the current recipe for large batch training (linear learning rate scaling with warm-up) is not general enough and training may diverge. To overcome this optimization difficulties we propose a new training algorithm based on Layer-wise Adaptive Rate Scaling (LARS). Using LARS, we scaled Alexnet up to a batch size of 8K, and Resnet-50 to a batch size of 32K without loss in accuracy.
核心贡献 · Key contributions
提出逐层自适应速率缩放（LARS）用于大规模批量训练。 Proposes Layer-wise Adaptive Rate Scaling (LARS) for large batch training.
展示 LARS 使 AlexNet 在批量大小 8K、ResNet-50 在 32K 下训练且无精度损失。 Shows LARS enables AlexNet training with batch size 8K and ResNet-50 with 32K without accuracy loss.
发现权重范数与梯度范数之比在各层间变化，从而启发逐层学习率。 Identifies that the ratio of weight norm to gradient norm varies across layers, motivating layer-wise learning rates.
证明批归一化有助于稳定大学习率训练，但单独使用不足。 Demonstrates that batch normalization helps stabilize large learning rate training but is insufficient alone.
表明极大批量的精度损失可通过延长训练来恢复。 Shows that accuracy loss for very large batches can be recovered by training longer.
提供经验证据表明泛化差距并非大批量精度损失的主因。 Provides empirical evidence that the generalization gap is not the main cause of accuracy loss for large batches.
局限 · Limitations
LARS 需要调整信任系数η，该系数可能因模型而异。 LARS requires tuning the trust coefficient η, which may vary across models.
实验仅限于 ImageNet 上的 AlexNet 和 ResNet-50，对其他架构的泛化性未知。 Experiments limited to AlexNet and ResNet-50 on ImageNet; generalizability to other architectures is unknown.
批量大小超过 32K 时，若不延长训练仍会出现精度损失。 Training with batch size above 32K still results in accuracy loss without longer training.
该方法未解决大批量训练本身的计算成本问题。 The method does not address the computational cost of large batch training itself.
由于数据增强有限和单裁剪测试，对比基线可能未反映最先进水平。 Comparison baselines may not reflect state-of-the-art due to limited data augmentation and single-crop testing.
论文章节 · Sections（共 8）
摘要Abstract
1 引言1 Introduction
2 背景2 Background
3 大批量训练 Alexnet 分析3 Analysis of Alexnet training with large batch