We explore a new class of diffusion models based on the transformer architecture. We train latent diffusion models of images, replacing the commonly-used U-Net backbone with a transformer that operates on latent patches. We analyze the scalability of our Diffusion Transformers (DiTs) through the lens of forward pass complexity as measured by Gflops. We find that DiTs with higher Gflops -- through increased transformer depth/width or increased number of input tokens -- consistently have lower FID. In addition to possessing good scalability properties, our largest DiT-XL/2 models outperform all prior diffusion models on the class-conditional ImageNet 512x512 and 256x256 benchmarks, achieving a state-of-the-art FID of 2.27 on the latter.
核心贡献 · Key contributions
提出扩散 Transformer（DiT），一种基于 Transformer 架构的新型扩散模型。 Introduces Diffusion Transformers (DiTs), a new class of diffusion models based on the transformer architecture.
用作用于潜在块的 Transformer 替代常用的 U-Net 主干，表明 U-Net 归纳偏置并非关键。 Replaces the commonly-used U-Net backbone with a transformer operating on latent patches, showing U-Net inductive bias is not crucial.
证明模型 Gflops 与样本质量（FID）之间存在强相关性，更高的 Gflops 持续带来更低的 FID。 Demonstrates strong correlation between model Gflops and sample quality (FID), with higher Gflops consistently yielding lower FID.
在类别条件 ImageNet 256x256 上实现 2.27 的 FID，超越所有先前扩散模型。 Achieves state-of-the-art FID of 2.27 on class-conditional ImageNet 256x256, outperforming all prior diffusion models.
表明 adaLN-Zero 条件机制计算效率最高，在四种块设计中质量最佳。 Shows adaLN-Zero conditioning mechanism is most compute-efficient and yields best quality among four block designs.
提升模型算力（Gflops）比参数量或采样算力对改进 DiT 性能更关键。 Scaling model compute (Gflops) is more critical than parameter count or sampling compute for improving DiT performance.
局限 · Limitations
仅在类别条件 ImageNet 的 256x256 和 512x512 分辨率上评估。 Only evaluated on class-conditional ImageNet at 256x256 and 512x512 resolutions.
使用 Stable Diffusion 的固定预训练 VAE；DiT 性能可能依赖 VAE 质量。 Uses a fixed pre-trained VAE from Stable Diffusion; DiT's performance may depend on VAE quality.
训练超参数主要沿用 ADM，未针对 DiT 进行广泛调优。 Training hyperparameters are largely retained from ADM without extensive tuning for DiT.
缩放分析仅限于 Gflops；未探索内存或延迟等其他复杂度指标。 Scaling analysis limited to Gflops; other complexity metrics like memory or latency not explored.
未涉及文本到图像生成或类别标签以外的其他条件模态。 Does not address text-to-image generation or other conditional modalities beyond class labels.
论文章节 · Sections（共 10）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3.1 预备知识3.1 Preliminaries
3.2 扩散 Transformer 设计空间3.2 Diffusion Transformer Design Space
4 实验设置4 Experimental Setup
5 实验5 Experiments
5.1 最先进的扩散模型5.1 State-of-the-Art Diffusion Models
5.2 模型规模与采样计算5.2 Scaling Model vs. Sampling Compute