Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, existing methods cannot maintain accuracy and hardware efficiency at the same time. We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation. SmoothQuant enables an INT8 quantization of both weights and activations for all the matrix multiplications in LLMs, including OPT, BLOOM, GLM, MT-NLG, Llama-1/2, Falcon, Mistral, and Mixtral models. We demonstrate up to 1.56x speedup and 2x memory reduction for LLMs with negligible loss in accuracy. SmoothQuant enables serving 530B LLM within a single node. Our work offers a turn-key solution that reduces hardware costs and democratizes LLMs. Code is available at https://github.com/mit-han-lab/smoothquant.
核心贡献 · Key contributions
提出 SmoothQuant，一种无需训练的后训练量化方法，实现大语言模型的 W8A8 量化。 Proposes SmoothQuant, a training-free post-training quantization method for LLMs enabling W8A8 quantization.
识别激活值异常值为量化主要难点，并通过离线迁移至权重来平滑它们。 Identifies activation outliers as key quantization difficulty and smooths them via offline migration to weights.
在多种大语言模型上实现高达 1.56 倍加速和 2 倍内存减少，精度损失可忽略。 Achieves up to 1.56x speedup and 2x memory reduction with negligible accuracy loss across multiple LLM families.
支持在单个 8-GPU 节点上部署 530B 参数的大语言模型。 Enables serving a 530B parameter LLM within a single 8-GPU node.
集成到 PyTorch 和 FasterTransformer 中，便于实际部署。 Integrates into PyTorch and FasterTransformer for practical deployment.
在 OPT、BLOOM、GLM、LLaMA 和 MT-NLG 等高达 530B 参数的模型上验证有效性。 Demonstrates effectiveness on OPT, BLOOM, GLM, LLaMA, and MT-NLG models up to 530B parameters.
局限 · Limitations
需要校准数据（512 个句子）来计算平滑因子和量化步长。 Requires calibration data (512 sentences) to compute smoothing factors and quantization steps.
超参数α需针对不同模型族调整（如 OPT/BLOOM 用 0.5，GLM 用 0.75）。 Hyperparameter α must be tuned per model family (e.g., 0.5 for OPT/BLOOM, 0.75 for GLM).
当校准与评估数据分布不同时，静态量化可能导致精度下降。 Static quantization may cause accuracy drop when calibration and evaluation distributions differ.
方法仅针对线性层和注意力层，其他操作仍保持 FP16。 Method focuses on linear and attention layers; other operations remain in FP16.
效果依赖于模型架构，某些模型（如 GLM）需要更强的平滑处理。 Effectiveness depends on model architecture; some models (e.g., GLM) require stronger smoothing.
论文章节 · Sections（共 13）
摘要Abstract
1 引言1 Introduction
2 预备知识2 Preliminaries
3 量化困难回顾3 Review of Quantization Difficulty
4 SmoothQuant4 SmoothQuant
5.1 实验设置5.1 Setups
5.2 精确量化5.2 Accurate Quantization
5.3 加速与内存节省5.3 Speedup and Memory Saving
5.4 扩展：单节点 530B 模型5.4 Scaling Up: 530B Model Within a Single Node