QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→量化可以加速大语言模型（LLM）推理。在 INT8 量化之外，研究社区正在积极探索更低精度，如 INT4。然而，最先进的 INT4 量化技术仅能加速低批量、边缘 LLM 推理，无法在高批量、基于云的 LLM 服务中带来性能提升。我们发现了一个关键问题：现有的 INT4 量化方法在 GPU 上对权重或部分和进行反量化时，会产生显著的运行时开销（20-90%）。为了解决这一挑战，我们引入了 QoQ，一种 W4A8KV4 量化算法，采用 4 位权重、8 位激活和 4 位 KV 缓存。QoQ 代表拉丁语中的 4-8-4。QoQ 由 QServe 推理库实现，并取得了实测加速。驱动 QServe 的关键洞察是，GPU 上 LLM 服务的效率受到低吞吐量 CUDA 核心操作的严重影响。基于这一洞察，在 QoQ 算法中，我们引入了渐进式量化，可以在 W4A8 GEMM 中实现低反量化开销。此外，我们开发了 SmoothAttention，以有效缓解 4 位 KV 量化带来的精度下降。在 QServe 系统中，我们执行计算感知的权重重排序，并利用寄存器级并行性来减少反量化延迟。我们还使融合注意力变为内存受限，从而利用 KV4 量化带来的性能提升。因此，与 TensorRT-LLM 相比，QServe 在 A100 上将 Llama-3-8B 的最大可达到服务吞吐量提升了 1.2 倍，在 L40S 上提升了 1.4 倍；在 A100 上将 Qwen1.5-72B 提升了 2.4 倍，在 L40S 上提升了 3.5 倍。值得注意的是，L40S GPU 上的 QServe 甚至可以达到比 A100 上的 TensorRT-LLM 更高的吞吐量。因此，QServe 有效地将 LLM 服务的美元成本降低了 3 倍。代码可在 https://github.com/mit-han-lab/omniserve 获取。
Quantization can accelerate large language model (LLM) inference. Going beyond INT8 quantization, the research community is actively exploring even lower precision, such as INT4. Nonetheless, state-of-the-art INT4 quantization techniques only accelerate low-batch, edge LLM inference, failing to deliver performance gains in large-batch, cloud-based LLM serving. We uncover a critical issue: existing INT4 quantization methods suffer from significant runtime overhead (20-90%) when dequantizing either weights or partial sums on GPUs. To address this challenge, we introduce QoQ, a W4A8KV4 quantization algorithm with 4-bit weight, 8-bit activation, and 4-bit KV cache. QoQ stands for quattuor-octo-quattuor, which represents 4-8-4 in Latin. QoQ is implemented by the QServe inference library that achieves measured speedup. The key insight driving QServe is that the efficiency of LLM serving on GPUs is critically influenced by operations on low-throughput CUDA cores. Building upon this insight, in QoQ algorithm, we introduce progressive quantization that can allow low dequantization overhead in W4A8 GEMM. Additionally, we develop SmoothAttention to effectively mitigate the accuracy degradation incurred by 4-bit KV quantization. In the QServe system, we perform compute-aware weight reordering and take advantage of register-level parallelism to reduce dequantization latency. We also make fused attention memory-bound, harnessing the performance gain brought by KV4 quantization. As a result, QServe improves the maximum achievable serving throughput of Llama-3-8B by 1.2x on A100, 1.4x on L40S; and Qwen1.5-72B by 2.4x on A100, 3.5x on L40S, compared to TensorRT-LLM. Remarkably, QServe on L40S GPU can achieve even higher throughput than TensorRT-LLM on A100. Thus, QServe effectively reduces the dollar cost of LLM serving by 3x. Code is available at https://github.com/mit-han-lab/omniserve.