零样本文本到图像生成

Zero-Shot Text-to-Image Generation

亚历克·拉德福德 Alec Radford · OpenAI · 2021-02-24 · arXiv:2102.12092 ↗ · 被引 6535

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

文本到图像生成传统上专注于在固定数据集上寻找更好的建模假设。这些假设可能涉及复杂的架构、辅助损失或训练期间提供的对象部件标签或分割掩码等辅助信息。我们描述了一种基于 transformer 的简单方法，该方法将文本和图像令牌作为单一数据流进行自回归建模。在足够的数据和规模下，我们的方法在零样本评估中与之前的领域特定模型具有竞争力。

Text-to-image generation has traditionally focused on finding better modeling assumptions for training on a fixed dataset. These assumptions might involve complex architectures, auxiliary losses, or side information such as object part labels or segmentation masks supplied during training. We describe a simple approach for this task based on a transformer that autoregressively models the text and image tokens as a single stream of data. With sufficient data and scale, our approach is competitive with previous domain-specific models when evaluated in a zero-shot fashion.

核心贡献 · Key contributions

• 提出一种基于 Transformer 的简单方法，通过将文本和图像令牌建模为单一数据流，实现零样本文本到图像生成。
  Proposes a simple transformer-based approach for zero-shot text-to-image generation by modeling text and image tokens as a single data stream.
• 证明扩展数据规模和模型规模（120 亿参数）可生成高保真、灵活的图像，与领域特定模型竞争。
  Demonstrates that scaling data and model size (12-billion parameters) yields high-fidelity, flexible image generation competitive with domain-specific models.
• 引入两阶段训练流程：离散 VAE 用于图像压缩，自回归 Transformer 用于联合文本-图像建模。
  Introduces a two-stage training procedure with a discrete VAE for image compression and an autoregressive transformer for joint text-image modeling.
• 在 MS-COCO 上实现最先进的零样本性能，人类评估者 90%的情况下偏好本方法。
  Achieves state-of-the-art zero-shot performance on MS-COCO, preferred by human evaluators 90% of the time over prior work.
• 展示单一生成模型涌现的零样本图像到图像翻译和组合泛化等能力。
  Shows emergent capabilities like zero-shot image-to-image translation and combinatorial generalization from a single generative model.
• 开发大规模 Transformer 稳定训练的实用技术，包括逐残差块梯度缩放和 PowerSGD 压缩。
  Develops practical techniques for stable training of large-scale transformers, including per-resblock gradient scaling and PowerSGD compression.

局限 · Limitations

• 通过 dVAE 的强图像压缩限制了高频细节生成，在 CUB 等专业数据集上降低 FID/IS。
  Heavy image compression via dVAE limits high-frequency detail generation, reducing FID/IS on specialized datasets like CUB.
• 在专业分布（如 CUB 鸟类）上的零样本性能显著差于领域特定模型。
  Zero-shot performance on specialized distributions (e.g., CUB birds) is significantly worse than domain-specific models.
• 模型需要巨大计算资源（120 亿参数、2.5 亿图像-文本对），限制可复现性。
  Model requires massive computational resources (12-billion parameters, 250M image-text pairs), limiting reproducibility.
• 在复杂组合任务（如变量绑定）上表现不一致。
  Inconsistent performance on complex compositional tasks like variable binding (e.g., hedgehog in sweater walking dog).
• 16 位精度训练不稳定，需要自定义梯度缩放和参数分片以避免下溢。
  Training instability in 16-bit precision required custom gradient scaling and parameter sharding to avoid underflow.

论文章节 · Sections（共 15）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2 方法 2 Method
• 2.1 第一阶段：学习视觉码本 2.1 Stage One: Learning the Visual Codebook
• 2.2 第二阶段：学习先验 2.2 Stage Two: Learning the Prior
• 2.3 数据收集 2.3 Data Collection
• 2.4 混合精度训练 2.4 Mixed-Precision Training
• 2.5 分布式优化 2.5 Distributed Optimization
• 2.6 样本生成 2.6 Sample Generation
• 3 实验 3 Experiments
• 3.1 定量结果 3.1 Quantitative Results
• 3.2 数据重叠分析 3.2 Data Overlap Analysis
• 3.3 定性发现 3.3 Qualitative Findings
• 4 结论 4 Conclusion
• 致谢 Acknowledgements

阅读逐段中英对照全文 →