We present Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. Imagen builds on the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-fidelity image generation. Our key discovery is that generic large language models (e.g. T5), pretrained on text-only corpora, are surprisingly effective at encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample fidelity and image-text alignment much more than increasing the size of the image diffusion model. Imagen achieves a new state-of-the-art FID score of 7.27 on the COCO dataset, without ever training on COCO, and human raters find Imagen samples to be on par with the COCO data itself in image-text alignment. To assess text-to-image models in greater depth, we introduce DrawBench, a comprehensive and challenging benchmark for text-to-image models. With DrawBench, we compare Imagen with recent methods including VQ-GAN+CLIP, Latent Diffusion Models, and DALL-E 2, and find that human raters prefer Imagen over other models in side-by-side comparisons, both in terms of sample quality and image-text alignment. See https://imagen.research.google/ for an overview of the results.
核心贡献 · Key contributions
Imagen 在 COCO 上实现了 7.27 的零样本 FID，优于先前方法。 Imagen achieves state-of-the-art zero-shot FID of 7.27 on COCO, outperforming prior methods.
引入 DrawBench，一个用于评估文本到图像模型的综合基准。 Introduces DrawBench, a comprehensive benchmark for evaluating text-to-image models.
表明扩大语言模型比扩大扩散模型更能提升图像质量。 Shows that scaling large language models improves image quality more than scaling diffusion models.
提出动态阈值处理以支持高引导权重，提升逼真度和对齐性。 Proposes dynamic thresholding for high guidance weights, improving photorealism and alignment.
证明冻结的 T5-XXL 文本编码器在人类评估中优于 CLIP。 Demonstrates that frozen T5-XXL text encoder outperforms CLIP in human evaluation.
在 COCO 上达到人类水平的图像-文本对齐，样本与参考图像评分相当。 Achieves human-level image-text alignment on COCO, with samples rated on par with reference images.
局限 · Limitations
Imagen 在生成逼真人物方面存在困难，偏好率较低。 Imagen struggles to generate photorealistic people, as shown by lower preference rates.
训练数据包含偏见和有害内容，限制了安全公开部署。 Training data contains biases and harmful content, limiting safe public deployment.
模型继承了语言模型的社会偏见，如肤色和性别刻板印象。 Model inherits social biases from language models, e.g., skin tone and gender stereotypes.
评估集中于英文提示；其他语言的表现未知。 Evaluation focuses on English prompts; performance on other languages is unknown.
由于大模型规模和级联架构，计算成本高。 High computational cost due to large model sizes and cascaded architecture.
论文章节 · Sections（共 15）
摘要Abstract
1 引言1 Introduction
2 Imagen 模型2 Imagen
2.1 预训练文本编码器2.1 Pretrained text encoders
2.2 扩散模型与无分类器引导2.2 Diffusion models and classifier-free guidance
2.3 大引导权重采样器2.3 Large guidance weight samplers
2.4 鲁棒级联扩散模型2.4 Robust cascaded diffusion models
2.5 神经网络架构2.5 Neural network architecture
3 文本到图像模型评估3 Evaluating Text-to-Image Models
4 实验4 Experiments
4.1 训练细节4.1 Training details
4.2 COCO 数据集结果4.2 Results on COCO
5 相关工作5 Related Work
6 结论、局限性与社会影响6 Conclusions, Limitations and Societal Impact