AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→大语言模型（LLM）已变革众多 AI 应用。设备端 LLM 日益重要：在边缘设备本地运行 LLM 可降低云计算成本并保护用户隐私。然而，巨大的模型规模和有限的硬件资源带来了显著的部署挑战。我们提出激活感知权重量化（AWQ），一种面向 LLM 低位仅权重量化的硬件友好方法。AWQ 发现并非 LLM 中所有权重同等重要。仅保护 1%的显著权重即可大幅降低量化误差。为识别显著权重通道，应参考激活分布而非权重。为避免硬件低效的混合精度量化，我们通过数学推导得出缩放显著通道可降低量化误差。AWQ 采用等价变换缩放显著权重通道以保护它们。缩放因子通过离线收集激活统计信息确定。AWQ 不依赖任何反向传播或重构，因此能泛化至不同领域和模态而不过拟合校准集。AWQ 在各种语言建模和领域特定基准（编码与数学）上优于现有工作。得益于更好的泛化性，它在指令调优语言模型上实现了出色的量化性能，并首次在多模态语言模型上取得优异效果。伴随 AWQ，我们实现了 TinyChat，一个专为 4 位设备端 LLM/VLM 设计的高效灵活推理框架。通过内核融合和平台感知权重打包，TinyChat 在桌面和移动 GPU 上相比 Huggingface FP16 实现实现了超过 3 倍的加速。它还使得 70B Llama-2 模型在移动 GPU 上的部署变得普及。
Large language models (LLMs) have transformed numerous AI applications. On-device LLM is becoming increasingly important: running LLMs locally on edge devices can reduce the cloud computing cost and protect users' privacy. However, the astronomical model size and the limited hardware resource pose significant deployment challenges. We propose Activation-aware Weight Quantization (AWQ), a hardware-friendly approach for LLM low-bit weight-only quantization. AWQ finds that not all weights in an LLM are equally important. Protecting only 1% salient weights can greatly reduce quantization error. To identify salient weight channels, we should refer to the activation distribution, not weights. To avoid the hardware-inefficient mix-precision quantization, we mathematically derive that scaling up the salient channels can reduce the quantization error. AWQ employs an equivalent transformation to scale the salient weight channels to protect them. The scale is determined by collecting the activation statistics offline. AWQ does not rely on any backpropagation or reconstruction, so it generalizes to different domains and modalities without overfitting the calibration set. AWQ outperforms existing work on various language modeling and domain-specific benchmarks (coding and math). Thanks to better generalization, it achieves excellent quantization performance for instruction-tuned LMs and, for the first time, multi-modal LMs. Alongside AWQ, we implement TinyChat, an efficient and flexible inference framework tailored for 4-bit on-device LLM/VLMs. With kernel fusion and platform-aware weight packing, TinyChat offers more than 3x speedup over the Huggingface FP16 implementation on both desktop and mobile GPUs. It also democratizes the deployment of the 70B Llama-2 model on mobile GPUs.