Low-precision training is considered an effective strategy for reducing both training and downstream inference costs. Previous scaling laws for precision mainly focus on integer quantization, which pay less attention to the constituents in floating-point (FP) quantization, and thus cannot well fit the LLM losses in this scenario. In contrast, while FP quantization training is more commonly implemented in production, it's research has been relatively superficial. In this paper, we thoroughly explore the effects of FP quantization targets, exponent bits, mantissa bits, and the calculation granularity of the scaling factor in FP quantization training performance of LLM models. In addition to an accurate FP quantization unified scaling law, we also provide valuable suggestions for the community: (1) Exponent bits contribute slightly more to the model performance than mantissa bits. We provide the optimal exponent-mantissa bit ratio for different bit numbers, which is available for future reference by hardware manufacturers; (2) We discover the formation of the critical data size in low-precision LLM training. Too much training data exceeding the critical data size will inversely bring in degradation of LLM performance; (3) The optimal FP quantization precision is directly proportional to the computational power, but within a wide computational power range. We estimate that the best cost-performance precision should lie between 4-8 bits.
核心贡献 · Key contributions
提出了浮点量化训练的统一缩放定律，整合了数据规模、模型规模、指数位、尾数位和块大小。 Proposes a unified scaling law for floating-point quantization training, integrating data size, model size, exponent, mantissa, and block size.
发现指数位对模型性能的贡献略大于尾数位，为硬件提供了最优的指尾位比。 Discovers that exponent bits contribute slightly more to model performance than mantissa bits, providing optimal bit ratios for hardware.
识别出低精度大语言模型训练中的临界数据量，超过该数据量后性能会下降。 Identifies a critical data size in low-precision LLM training, beyond which performance degrades.
估计在广泛的计算预算范围内，最优性价比精度介于 4-8 比特之间。 Estimates the optimal cost-performance precision lies between 4-8 bits for a wide range of compute budgets.
在 12 亿参数模型上验证了缩放定律，显示出准确的损失预测。 Validates the scaling law on 1.2B parameter models, showing accurate loss prediction.
为实际大语言模型训练提供了关于量化目标和块大小影响的见解。 Provides insights on quantization targets and block size effects for practical LLM training.
局限 · Limitations
实验仅限于 LLaMA 架构，未验证对其他架构（如 Mamba）的适用性。 Experiments limited to LLaMA architecture; applicability to other architectures like Mamba not verified.
缩放定律基于最大 6.79 亿参数的模型推导，更大规模可能需要进一步验证。 Scaling law derived from models up to 679M parameters; larger scales may require further validation.
专注于经典浮点量化，未涵盖更新的低位量化方法。 Focuses on classical floating-point quantization; newer low-bit methods not covered.
模拟的浮点格式可能无法完全反映硬件行为。 Simulated floating-point formats may not fully reflect hardware behavior.
临界数据量和最优精度估计依赖于拟合参数，可能因数据集不同而变化。 Critical data size and optimal precision estimates depend on fitted parameters; may vary with different datasets.
论文章节 · Sections（共 17）
摘要Abstract
1 引言1 Introduction
2 预备知识2 Preliminary
3.1 方法与实现3.1 Method and Implementation
3.2 实验设置3.2 Setup
3.3 基本缩放律形式3.3 Basic Scaling Law Form
3.4 量化目标3.4 Quantization Targets
3.5 指数与尾数3.5 Exponent and Mantissa
3.6 缩放因子的块大小3.6 Block Size of Scaling Factor
4 浮点量化训练的统一缩放律4 A Unified Scaling Law for Floating–Point Quantization Training