We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with "zero convolutions" (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, eg, edges, depth, segmentation, human pose, etc, with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small (<50k) and large (>1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.
核心贡献 · Key contributions
提出 ControlNet，一种向大型预训练文本到图像扩散模型添加空间条件控制的神经网络架构。 Proposes ControlNet, a neural network architecture for adding spatial conditioning controls to large pretrained text-to-image diffusion models.
使用零初始化卷积层连接可训练副本和锁定副本，防止微调期间的有害噪声。 Uses zero-initialized convolution layers to connect trainable and locked copies, preventing harmful noise during finetuning.
展示了在小于 5 万到大于 100 万样本的数据集上的稳健训练，避免了过拟合和灾难性遗忘。 Demonstrates robust training with datasets ranging from <50k to >1m samples, avoiding overfitting and catastrophic forgetting.
支持多种条件输入（边缘、深度、分割、人体姿态），可单条件或多条件，带或不带提示。 Supports diverse conditioning inputs (edges, depth, segmentation, human pose) with single or multiple conditions, with or without prompts.
在深度到图像任务中，使用有限资源（单 GPU、20 万样本）达到与工业模型竞争的结果。 Achieves competitive results with industrial models using limited resources (single GPU, 200k samples) for depth-to-image tasks.
支持多个 ControlNet 的组合而无需额外加权，便于复杂条件控制。 Enables composition of multiple ControlNets without extra weighting, facilitating complex conditional control.
局限 · Limitations
ControlNet 需要预训练扩散模型（如 Stable Diffusion）作为骨干，限制了在其他架构上的适用性。 ControlNet requires a pretrained diffusion model (e.g., Stable Diffusion) as backbone, limiting applicability to other architectures.
为新条件训练 ControlNet 需要特定任务数据集，可能规模小且难以收集。 Training ControlNet for new conditions requires task-specific datasets, which may be small and hard to collect.
突然收敛现象可能导致学习突变，可能错过渐进改进。 The sudden convergence phenomenon may cause abrupt learning, potentially missing gradual improvements.
ControlNet 的性能依赖于条件输入的质量；噪声或不准确的输入会降低结果。 ControlNet's performance depends on the quality of conditioning inputs; noisy or inaccurate inputs degrade results.
该方法主要在 Stable Diffusion 上测试，对其他扩散模型的泛化性尚未充分验证。 The method has been tested primarily on Stable Diffusion; generalization to other diffusion models is not fully validated.
论文章节 · Sections（共 17）
摘要Abstract
1 引言1 Introduction
2.1 微调神经网络2.1 Finetuning Neural Networks
2.2 图像扩散2.2 Image Diffusion
2.3 图像到图像翻译2.3 Image-to-Image Translation
3 方法3 Method
3.1 ControlNet3.1 ControlNet
3.2 用于文本到图像扩散的 ControlNet3.2 ControlNet for Text-to-Image Diffusion