We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.
核心贡献 · Key contributions
提出三种方法确定算力最优的模型规模与训练词元数，发现二者应等比例缩放。 Proposes three methods to determine compute-optimal model size and training tokens, showing they should scale equally.
训练超过 400 个模型（参数从 7000 万到 160 亿），经验性地推导缩放定律。 Trains over 400 models from 70M to 16B parameters to empirically derive scaling laws.
证明当前大语言模型因过度关注模型规模而严重训练不足。 Demonstrates that current large language models are significantly undertrained due to overemphasis on model size.
引入 Chinchilla（70B 参数，1.4 万亿词元训练），在多数任务上超越 Gopher、GPT-3 等更大模型。 Introduces Chinchilla, a 70B parameter model trained on 1.4T tokens, outperforming larger models like Gopher and GPT-3.
在 MMLU 上达到 67.5%准确率，比 Gopher 提升 7%。 Achieves state-of-the-art 67.5% accuracy on MMLU, a 7% improvement over Gopher.
算力最优训练降低推理和微调成本，便于下游应用。 Shows compute-optimal training reduces inference and fine-tuning costs, facilitating downstream usage.
局限 · Limitations
仅比较了两次大规模训练（Chinchilla 和 Gopher），中间规模验证不足。 Only two large-scale training runs (Chinchilla and Gopher) are compared, limiting validation at intermediate scales.
假设算力、模型规模和词元数呈幂律关系，但在高算力预算下观察到凹性。 Assumes a power-law relationship between compute, model size, and tokens, but observes concavity at high compute budgets.
所有训练均使用不到一个 epoch 的数据，未探索多 epoch 场景。 All training runs use less than one epoch of data; multiple-epoch regime is not explored.
分析依赖平滑训练损失作为测试损失的代理，可能未完全反映泛化能力。 Analysis relies on smoothed training loss as a proxy for test loss, which may not fully capture generalization.
承认更大数据集带来的伦理和隐私问题，但未充分解决。 Ethical and privacy concerns from larger datasets are acknowledged but not fully addressed.
论文章节 · Sections（共 13）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 估计最优参数/训练令牌分配3 Estimating the optimal parameter/training tokens allocation
3.1 方法 1：固定模型大小，改变训练令牌数量3.1 Approach 1: Fix model sizes and vary number of training tokens