We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.
核心贡献 · Key contributions
语言模型交叉熵损失随模型规模、数据集大小和算力呈幂律缩放，跨越七个数量级。 Language model cross-entropy loss follows power-law scaling with model size, dataset size, and compute over seven orders of magnitude.
最优算力高效训练使用非常大的模型在适度数据上训练，并在收敛前停止。 Optimal compute-efficient training uses very large models trained on modest data, stopping before convergence.
与规模因素相比，性能对深度或宽度等架构细节的依赖很弱。 Performance depends weakly on architectural details like depth or width compared to scale factors.
过拟合和训练速度由模型大小和数据集大小的简单方程控制。 Overfitting and training speed are governed by simple equations relating model size and dataset size.
更大的模型样本效率更高，达到相同损失所需的数据点更少。 Larger models are more sample-efficient, requiring fewer data points to reach the same loss.
训练的关键批次大小随损失呈幂律变化，与模型大小无关。 The critical batch size for training follows a power-law in the loss, independent of model size.
局限 · Limitations
由于自然语言的非零熵，缩放定律可能在极端规模下失效。 Scaling laws may break down at extreme scales due to non-zero entropy of natural language.
结果基于 Transformer 架构，可能不适用于其他架构。 Results are based on Transformer architecture; may not generalize to other architectures.
研究使用单一数据集（WebText2）；迁移到其他分布会带来恒定的损失惩罚。 The study uses a single dataset (WebText2); transfer to other distributions incurs a constant loss penalty.
最优算力分配假设固定批次大小，可能不适用于变化的硬件约束。 Optimal compute allocation assumes fixed batch size and may not hold for varying hardware constraints.
幂律拟合是经验性的，缺乏第一性原理的理论推导。 The power-law fits are empirical and lack a theoretical derivation from first principles.
论文章节 · Sections（共 43）
摘要Abstract
1 引言1 Introduction
1.1 总结1.1 Summary
1.2 缩放定律总结1.2 Summary of Scaling Laws
1.3 符号说明1.3 Notation
2 背景与方法2 Background and Methods
2.1 Transformer 的参数与算力缩放2.1 Parameter and Compute Scaling of Transformers
2.2 训练流程2.2 Training Procedures
2.3 数据集2.3 Datasets
3 实证结果与基本幂律3 Empirical Results and Basic Power Laws
3.1 近似 Transformer 形状与超参数独立性3.1 Approximate Transformer Shape and Hyperparameter Independence
3.2 非嵌入参数计数 N 下的性能3.2 Performance with Non-Embedding Parameter Count N𝑁N
3.3 数据集大小与算力下的性能3.3 Performance with Dataset Size and Compute
4 描绘无限数据极限与过拟合4 Charting the Infinite Data Limit and Overfitting