This article reviews the empirical scaling laws in deep learning, which describe how training loss decreases predictably with increases in model size, dataset size, and compute, following power-law relationships. It traces the historical development from early theoretical work by Amari et al. and empirical studies by Hestness et al., through the influential Kaplan et al. scaling laws for language models, to the Chinchilla paper that revised the compute-optimal allocation. The core debate centers on how to optimally allocate compute between model parameters and training tokens: Kaplan et al. suggested model size should grow faster than data, while Chinchilla, using more careful experiments and three complementary methods, concluded that model size and data should scale at equal rates, implying that many existing large models were undertrained. The article highlights that scaling laws provide a practical framework for predicting performance and guiding resource allocation in large-scale model training, with the key takeaway being that the optimal balance between model size and data is crucial for efficiency.
核心贡献 · Key contributions
综述深度学习中的经验缩放定律，追溯从早期理论工作到 Kaplan 等人和 Chinchilla 的发展，强调模型规模与数据之间算力分配的核心争论。 Reviews empirical scaling laws in deep learning, tracing development from early theoretical work to Kaplan et al. and Chinchilla, highlighting the core debate on compute allocation between model size and data.
详述 Kaplan 等人关于模型规模应比数据增长更快的发现，以及 Chinchilla 修正的结论，即模型规模和数据应等速缩放，暗示许多大型模型训练不足。 Details Kaplan et al.'s finding that model size should grow faster than data, and Chinchilla's revised conclusion that model size and data should scale equally, implying many large models were undertrained.
提出 Chinchilla 中三种互补的缩放定律拟合方法，它们在计算最优前沿上达成一致，为预测性能和指导资源分配提供了令人信服的框架。 Presents three complementary methods in Chinchilla for fitting scaling laws, which agree on compute-optimal frontier, providing a convincing framework for predicting performance and guiding resource allocation.
讨论数据受限的缩放，表明重复数据的价值呈指数衰减，并引入基于容量比和重复次数的过拟合惩罚，在有限数据下改善拟合。 Discusses data-constrained scaling, showing that repeated data's value decays exponentially, and introduces overfitting penalties based on capacity ratio and repetition count, improving fit under limited data.
强调缩放定律拟合对参数计数、精度和拟合区域等程序性选择的敏感性，以 Kaplan 与 Chinchilla 的分歧作为外推陷阱的例子。 Highlights sensitivity of scaling law fitting to procedural choices like parameter counting, precision, and fit region, using Kaplan vs. Chinchilla disagreement as an example of extrapolation pitfalls.
局限 · Limitations
缩放定律假设无限唯一数据且无重复；当数据受限时可能失效，需要调整如有效数据和过拟合惩罚。 Scaling laws assume unlimited unique data and no repetition; they may fail when data is constrained, requiring adjustments like effective data and overfitting penalties.
拟合对程序性选择敏感，导致不同结论；损失精度或拟合区域的微小变化可显著改变外推预测。 Fitting is sensitive to procedural choices, leading to different conclusions; small changes in loss precision or fit region can alter extrapolated predictions significantly.
幂律缩放的理论解释不完整；数据流形划分或技能量化等假设仍未验证，难以实证检验。 Theoretical explanations for power-law scaling are incomplete; hypotheses like data manifold partitioning or skill quantization remain unverified and hard to test empirically.
缩放定律假设架构、优化器和数据管道不变；这些因素的变化可能使预测失效，如训练不足的模型导致不同结论。 Scaling laws assume constant architecture, optimizer, and data pipeline; changes in these factors may invalidate predictions, as seen in undertrained models leading to different conclusions.
数据重复效应是经验建模，但确切形式和参数必要性仍不清楚；模型可能低估高重复轮次失败模型的损失。 Data repetition effects are modeled empirically, but exact forms and parameter necessity remain unclear; models may underestimate loss for failing models with high repetition epochs.
论文章节 · Sections（共 12）
概述Overview
早期：机器学习损失的可预测性Early days: ML loss predictability
Kaplan 等人的缩放定律Kaplan et al.’s Scaling Laws
Chinchilla 缩放定律Chinchilla Scaling Laws
方法 1：固定模型规模，变化词元预算Method 1: Fix model sizes, vary the token budget
方法 2：IsoFLOP 曲线Method 2: IsoFLOP profiles
方法 3：参数拟合Method 3: Parametric fit
调和 Kaplan 与 ChinchillaReconciling Kaplan and Chinchilla
为什么是幂律？Why power law?
数据受限区域的缩放定律Scaling Laws in Data-Limited Region
现实中拟合缩放定律的棘手之处Trickiness of Fitting Scaling Laws in Reality