We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal image$\leftrightarrow$text models, and mathematical problem solving. In all cases autoregressive Transformers smoothly improve in performance as model size and compute budgets increase, following a power-law plus constant scaling law. The optimal model size also depends on the compute budget through a power-law, with exponents that are nearly universal across all data domains. The cross-entropy loss has an information theoretic interpretation as $S($True$) + D_{\mathrm{KL}}($True$||$Model$)$, and the empirical scaling laws suggest a prediction for both the true data distribution's entropy and the KL divergence between the true and model distributions. With this interpretation, billion-parameter Transformers are nearly perfect models of the YFCC100M image distribution downsampled to an $8\times 8$ resolution, and we can forecast the model size needed to achieve any given reducible loss (ie $D_{\mathrm{KL}}$) in nats/image for other resolutions. We find a number of additional scaling laws in specific domains: (a) we identify a scaling relation for the mutual information between captions and images in multimodal models, and show how to answer the question "Is a picture worth a thousand words?"; (b) in the case of mathematical problem solving, we identify scaling laws for model performance when extrapolating beyond the training distribution; (c) we finetune generative image models for ImageNet classification and find smooth scaling of the classification loss and error rate, even as the generative loss levels off. Taken together, these results strengthen the case that scaling laws have important implications for neural network performance, including on downstream tasks.
核心贡献 · Key contributions
在图像、视频、多模态和数学四个领域识别出交叉熵损失的经验缩放定律。 Identifies empirical scaling laws for cross-entropy loss across four domains: image, video, multimodal, and math.
表明最优模型大小随算力预算呈幂律缩放，指数约 0.7，跨领域通用。 Shows that optimal model size scales as a power-law with compute budget, with exponent ~0.7 across all domains.
将损失解释为熵加 KL 散度，可预测不可约损失及达到目标可约损失所需的模型大小。 Interprets loss as entropy plus KL divergence, enabling prediction of irreducible loss and model size for target reducible loss.
展示生成式图像模型微调用于分类时，损失和错误率随模型大小平滑缩放。 Demonstrates that generative image models finetuned for classification show smooth scaling of loss and error rate with model size.
发现多模态模型中互信息的缩放定律，量化图像与文本间的信息增益。 Finds scaling laws for mutual information in multimodal models, quantifying information gain between images and text.
表明数学问题外推性能主要取决于训练分布损失，而非模型大小。 Shows that math problem extrapolation performance depends mainly on training distribution loss, not model size.
局限 · Limitations
缩放定律在极大尺度下可能因优化效率或数据限制而失效。 Scaling laws may break down at very large scales due to optimization inefficiencies or data constraints.
结果仅限于自回归 Transformer 模型；其他架构可能表现出不同缩放行为。 Results are limited to autoregressive Transformer models; other architectures may exhibit different scaling.
将不可约损失解释为熵假设了无限数据和算力极限，实际中未完全实现。 The interpretation of irreducible loss as entropy assumes infinite data and compute limits, which are not fully realized.
将缩放趋势外推至测试范围之外可能不可靠，尤其对于下游任务。 Extrapolation of scaling trends beyond tested ranges may be unreliable, especially for downstream tasks.
互信息估计可能因空白标题或图像等分布外输入而产生偏差。 Mutual information estimates may be biased due to out-of-distribution inputs like blank captions or images.
论文章节 · Sections（共 30）
摘要Abstract
1 引言1 Introduction
1.1 结果概述1.1 Summary of Results
2 各领域中的核心经验缩放定律2 Central Empirical Scaling Laws in Each Domain
2.1 领域描述与训练设置2.1 Domain Descriptions and Training Setups
2.2 模型规模缩放与宽高比2.2 Model Size Scaling and Aspect Ratios
2.3 计算量缩放与最优模型规模2.3 Compute Scaling and Optimal Model Sizes
2.4 上下文中的损失位置取决于数据结构2.4 Loss versus Position in the Context Depends on the Structure of the Data
3 图像与视频建模、可约损失及下游任务3 Image and Video Modeling, the Reducible Loss, and Downstream Tasks
3.1 改变图像分辨率与编码3.1 Varying the Image Resolution and Encoding
3.2 视频建模与单帧处理3.2 Video Modeling and Individual Frames
3.3 单张图像的缩放趋势3.3 Scaling Trends for Individual Images
3.4 在 32x32 分辨率下对 ImageNet 进行微调3.4 Finetuning on ImageNet at 32x32 Resolution
4 多模态模型与信息增益4 Multimodal Models and Information Gain
5 数学问题求解与外推5 Mathematical Problem Solving and Extrapolation
6 计算量与数据量缩放定律中的不一致性6 An Inconsistency in Compute and Datasize Scaling Laws
7 相关工作7 Related Work
8 讨论8 Discussion
致谢Acknowledgments
贡献Contributions
附录 A 图像建模更多细节Appendix A More Details on Image Modeling
B.1 程序化生成的训练数据B.1 Procedurally Generated Training Data
B.2 数据集规模缩放B.2 Dataset Size Scaling
B.3 额外数学结果B.3 Additional Math Results
附录 C 额外多模态结果Appendix C Additional Multimodal Results
附录 D 额外语言结果Appendix D Additional Language Results
附录 E 互信息、信息增益与缩放Appendix E Mutual Information, Infogain, and Scaling
E.1 缩放关系的近似推导E.1 Approximate Derivation of Scaling Relations
E.2 估计真实世界分布间的 DKLE.2 Estimating DKLsubscript𝐷KLD_{\mathrm{KL}} Between Real-World Distributions