Diffusion models currently dominate the field of data-driven image synthesis with their unparalleled scaling to large datasets. In this paper, we identify and rectify several causes for uneven and ineffective training in the popular ADM diffusion model architecture, without altering its high-level structure. Observing uncontrolled magnitude changes and imbalances in both the network activations and weights over the course of training, we redesign the network layers to preserve activation, weight, and update magnitudes on expectation. We find that systematic application of this philosophy eliminates the observed drifts and imbalances, resulting in considerably better networks at equal computational complexity. Our modifications improve the previous record FID of 2.41 in ImageNet-512 synthesis to 1.81, achieved using fast deterministic sampling. As an independent contribution, we present a method for setting the exponential moving average (EMA) parameters post-hoc, i.e., after completing the training run. This allows precise tuning of EMA length without the cost of performing several training runs, and reveals its surprising interactions with network architecture, training time, and guidance.
核心贡献 · Key contributions
识别并修正常用于 ADM 扩散模型的激活和权重幅度失控问题，在 ImageNet-512 上实现了新的最佳 FID 1.81。 Identifies and rectifies uncontrolled activation and weight magnitude drifts in ADM diffusion models, achieving new SOTA FID 1.81 on ImageNet-512.
提出事后 EMA 方法，可在训练完成后调整 EMA 参数而无需重新训练，揭示了 EMA 与架构、训练时间及引导之间的相互作用。 Proposes post-hoc EMA to tune EMA parameters after training without retraining, revealing interactions with architecture, training time, and guidance.
系统性的幅度保持设计消除了训练过程中的漂移和不平衡，在相同计算代价下获得更好的网络。 Systematic magnitude-preserving design eliminates training drifts and imbalances, yielding better networks at equal computational cost.
发现使用很小的无条件模型即可实现无分类器引导，将近乎 50%的引导计算开销降低。 Shows that a small unconditional model suffices for classifier-free guidance, reducing computational overhead of guidance by nearly 50%.
在 ImageNet-64 上将确定性采样 FID 从 2.22 提升至 1.58（相似复杂度），通过缩放进一步降至 1.33。 Improves deterministic FID on ImageNet-64 from 2.22 to 1.58 at similar complexity, and further to 1.33 via scaling.
局限 · Limitations
修改针对 ADM U-Net 设计，是否适用于其他扩散架构（如 RIN、DiT）未经验证。 Modifications tailored to ADM U-Net; applicability to other diffusion architectures like RIN or DiT is not verified.
事后 EMA 方法需要在训练过程中存储多个权重快照，增加了存储开销。 Post-hoc EMA requires storing multiple weight snapshots during training, increasing storage overhead.
幅度保持设计依赖于统计假设（特征互不相关、等方差），这些假设可能不完全成立。 Magnitude-preserving design relies on statistical assumptions (uncorrelated features, equal variance) that may not hold perfectly.
Dropout 对大型模型有益，但对小型模型有害，引入了额外的超参数依赖。 Dropout is beneficial for large models but harmful for small ones, creating an additional hyperparameter dependency.
最优 EMA 长度在 FID 和基于 DINOv2 的 Fréchet 距离之间不一致，表明指标敏感性不同。 Optimal EMA length differs between FID and DINOv2-based Fréchet distance, indicating metric-dependent sensitivity.