SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→扩散模型能有效生成高质量图像，但随着规模扩大，内存需求增加和延迟升高带来了部署挑战。本文旨在通过将权重和激活量化至 4 比特来加速扩散模型。在如此激进的量化级别下，权重和激活都高度敏感，现有的训练后量化方法（如平滑）已不足够。为克服这一限制，我们提出 SVDQuant，一种新的 4 比特量化范式。与在权重和激活之间重新分配异常值的平滑方法不同，我们的方法使用低秩分支吸收这些异常值。我们首先将异常值从激活转移到权重中，然后使用高精度低秩分支通过奇异值分解（SVD）处理权重异常值，而低比特量化分支处理残差。这一过程减轻了双方的量化难度。然而，独立运行低秩分支会因额外的激活数据移动而产生显著开销，抵消量化加速。为解决此问题，我们协同设计了推理引擎 Nunchaku，将低秩分支的内核融合到低比特分支的内核中，以消除冗余内存访问。该引擎还能无缝支持现成的低秩适配器（LoRA），无需重新量化。在 SDXL、PixArt-Σ和 FLUX.1 上的大量实验验证了 SVDQuant 在保持图像质量方面的有效性。我们将 12B FLUX.1 模型的内存使用减少了 3.5 倍，在配备 INT4 精度的 16GB 笔记本 4090 GPU 上，相比 4 比特仅权重量化（W4A16）基线实现了 3.0 倍加速。在最新的 Blackwell 架构 RTX 5090 桌面 GPU 上，使用 NVFP4 精度，相比 W4A16 模型实现了 3.1 倍加速。
Diffusion models can effectively generate high-quality images. However, as they scale, rising memory demands and higher latency pose substantial deployment challenges. In this work, we aim to accelerate diffusion models by quantizing their weights and activations to 4 bits. At such an aggressive level, both weights and activations are highly sensitive, where existing post-training quantization methods like smoothing become insufficient. To overcome this limitation, we propose SVDQuant, a new 4-bit quantization paradigm. Different from smoothing, which redistributes outliers between weights and activations, our approach absorbs these outliers using a low-rank branch. We first consolidate the outliers by shifting them from activations to weights. Then, we use a high-precision, low-rank branch to take in the weight outliers with Singular Value Decomposition (SVD), while a low-bit quantized branch handles the residuals. This process eases the quantization on both sides. However, naively running the low-rank branch independently incurs significant overhead due to extra data movement of activations, negating the quantization speedup. To address this, we co-design an inference engine Nunchaku that fuses the kernels of the low-rank branch into those of the low-bit branch to cut off redundant memory access. It can also seamlessly support off-the-shelf low-rank adapters (LoRAs) without re-quantization. Extensive experiments on SDXL, PixArt-$Σ$, and FLUX.1 validate the effectiveness of SVDQuant in preserving image quality. We reduce the memory usage for the 12B FLUX.1 models by 3.5$\times$, achieving 3.0$\times$ speedup over the 4-bit weight-only quantization (W4A16) baseline on the 16GB laptop 4090 GPU with INT4 precision. On the latest RTX 5090 desktop with Blackwell architecture, we achieve a 3.1$\times$ speedup compared to the W4A16 model using NVFP4 precision.