In an increasing number of domains it has been demonstrated that deep learning models can be trained using relatively large batch sizes without sacrificing data efficiency. However the limits of this massive data parallelism seem to differ from domain to domain, ranging from batches of tens of thousands in ImageNet to batches of millions in RL agents that play the game Dota 2. To our knowledge there is limited conceptual understanding of why these limits to batch size differ or how we might choose the correct batch size in a new domain. In this paper, we demonstrate that a simple and easy-to-measure statistic called the gradient noise scale predicts the largest useful batch size across many domains and applications, including a number of supervised learning datasets (MNIST, SVHN, CIFAR-10, ImageNet, Billion Word), reinforcement learning domains (Atari and Dota), and even generative model training (autoencoders on SVHN). We find that the noise scale increases as the loss decreases over a training run and depends on the model size primarily through improved model performance. Our empirically-motivated theory also describes the tradeoff between compute-efficiency and time-efficiency, and provides a rough model of the benefits of adaptive batch-size training.
核心贡献 · Key contributions
提出梯度噪声尺度作为简单统计量，用于预测跨不同领域的最大有用批量大小。 Proposes gradient noise scale as a simple statistic to predict the largest useful batch size across diverse domains.
推导出将噪声尺度与大批量训练中计算-时间权衡曲线联系起来的理论框架。 Derives a theoretical framework linking noise scale to the compute-time tradeoff curve in large-batch training.
在包括监督学习、强化学习和生成模型的 8 个任务上实证验证了预测。 Empirically validates predictions across 8 tasks including supervised learning, RL, and generative models.
表明噪声尺度在训练过程中增加，复杂任务中更大，且与模型大小弱相关。 Shows noise scale increases during training and is larger for complex tasks, with weak dependence on model size.
提供了一种在数据并行训练中以最小开销测量噪声尺度的实用方法。 Provides a practical method to measure noise scale with minimal overhead in data-parallel training.
展示了基于噪声尺度的自适应批量大小训练的效率提升。 Demonstrates efficiency gains from adaptive batch-size training based on noise scale.
局限 · Limitations
理论依赖于局部二次近似，可能无法捕捉全局优化动态。 Theory relies on local quadratic approximation and may not capture global optimization dynamics.
噪声尺度预测精度因任务而异；与临界批量大小的比率可能相差一个数量级。 Noise scale prediction accuracy varies across tasks; ratio to critical batch size can differ by an order of magnitude.
假设学习率调优最优；噪声尺度通过训练温度依赖于学习率。 Assumes optimal learning rate tuning; noise scale depends on learning rate via training temperature.
未解决泛化差距问题；仅关注训练损失优化。 Does not address generalization gap; focuses solely on training loss optimization.
实证验证限于特定架构和优化器；可能无法推广到所有设置。 Empirical validation limited to specific architectures and optimizers; may not generalize to all settings.
论文章节 · Sections（共 28）
摘要Abstract
1 引言1 Introduction
2.1 直观图景2.1 Intuitive Picture
2.2 梯度、批次与梯度噪声尺度2.2 Gradients, Batches, and the Gradient Noise Scale
2.3 数据/时间效率权衡的预测2.3 Predictions for Data/Time Efficiency Tradeoffs
2.4 假设与注意事项2.4 Assumptions and Caveats
2.5 噪声尺度的预期模式2.5 Expected Patterns in the Noise Scale
2.6 小结2.6 Summary
3 实验3 Experiments
3.1 测量量3.1 Quantities Measured
3.2 结果3.2 Results
4 相关工作4 Related Work
5 讨论5 Discussion
致谢Acknowledgements
A.1 无开销的简单噪声尺度无偏估计A.1 Unbiased Estimate of the Simple Noise Scale with No Overhead
A.2 批次大小的系统搜索A.2 Systematic Searches Over Batch Sizes
A.3 帕累托前沿测量A.3 Pareto Front Measurements
A.4 学习任务细节A.4 Details of Learning Tasks
附录 B 所有任务的结果Appendix B Results for All Tasks
附录 C 温度与噪声尺度Appendix C Temperature and the Noise Scale
温度的玩具模型A Toy Model for the Temperature
附录 D 动态调整批次大小Appendix D Dynamically Varying the Batch Size
D.1 理论D.1 Theory
D.2 SVHN 案例研究D.2 An SVHN Case Study
附录 E 关于优化的评论Appendix E Comments on Optimization
E.1 确定性训练表现不佳E.1 Deterministic Training Performs Poorly
E.2 学习率缩放规则的动机E.2 Motivations for Learning Rate Scaling Rules
E.3 泛化性的初步测试E.3 Preliminary Tests of Generalization