We propose spatially-adaptive normalization, a simple but effective layer for synthesizing photorealistic images given an input semantic layout. Previous methods directly feed the semantic layout as input to the deep network, which is then processed through stacks of convolution, normalization, and nonlinearity layers. We show that this is suboptimal as the normalization layers tend to ``wash away'' semantic information. To address the issue, we propose using the input layout for modulating the activations in normalization layers through a spatially-adaptive, learned transformation. Experiments on several challenging datasets demonstrate the advantage of the proposed method over existing approaches, regarding both visual fidelity and alignment with input layouts. Finally, our model allows user control over both semantic and style. Code is available at https://github.com/NVlabs/SPADE .
核心贡献 · Key contributions
提出空间自适应归一化（SPADE），在归一化层中保留语义信息。 Proposes spatially-adaptive normalization (SPADE) to preserve semantic information in normalization layers.
在 COCO-Stuff、ADE20K 和 Cityscapes 数据集上取得最优结果。 Achieves state-of-the-art results on COCO-Stuff, ADE20K, and Cityscapes datasets.
通过使用随机噪声作为生成器输入，实现多模态合成。 Enables multi-modal synthesis by using random noise as generator input.
通过图像编码器支持风格引导的图像合成。 Supports style-guided image synthesis via an image encoder.
简化生成器架构，移除编码器，减少参数量。 Simplifies generator architecture by removing encoder, reducing parameters.
证明 SPADE 在语义条件化上优于特征拼接。 Demonstrates that SPADE outperforms feature concatenation for semantic conditioning.
局限 · Limitations
需要配对的语义分割掩码和图像进行训练。 Requires paired segmentation masks and images for training.
性能依赖于输入分割掩码的质量。 Performance depends on quality of input segmentation masks.
多模态合成受限于训练数据的多样性。 Multi-modal synthesis limited by diversity of training data.
由于多尺度判别器和谱归一化，计算成本高。 Computationally expensive due to multi-scale discriminators and spectral normalization.
风格迁移可能无法泛化到训练分布之外的未见风格。 Style transfer may not generalize to unseen styles outside training distribution.