Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.
核心贡献 · Key contributions
提出一个两阶段模型，包括从文本生成 CLIP 图像嵌入的先验模型和从嵌入生成图像的解码器。 Proposes a two-stage model with a prior generating CLIP image embeddings from text and a decoder generating images from embeddings.
表明显式生成图像表示可在几乎不损失真实感和标题相似度的情况下提高多样性。 Shows that generating image representations explicitly improves diversity with minimal loss in photorealism and caption similarity.
利用 CLIP 的联合嵌入空间，以零样本方式实现语言引导的图像操作。 Demonstrates language-guided image manipulations in a zero-shot manner using CLIP's joint embedding space.
比较自回归先验和扩散先验，发现扩散先验计算效率更高且生成样本质量更高。 Compares autoregressive and diffusion priors, finding diffusion priors more compute-efficient and producing higher-quality samples.
在 MS-COCO 验证集上实现了 10.39 的最先进零样本 FID 分数。 Achieves state-of-the-art zero-shot FID of 10.39 on MS-COCO validation set.
引入二分潜变量表示，支持图像变体、插值和文本差异操作。 Introduces a bipartite latent representation enabling image variations, interpolation, and text diffs.
局限 · Limitations
unCLIP 在将属性绑定到对象方面存在困难，例如将颜色分配给特定立方体。 unCLIP struggles with binding attributes to objects, e.g., assigning colors to specific cubes.
该模型在图像内生成连贯文本方面存在困难。 The model has difficulty generating coherent text within images.
由于 64x64 基础分辨率和上采样，复杂场景缺乏精细细节。 Complex scenes lack fine details due to the 64x64 base resolution and upsampling.
CLIP 嵌入可能无法精确编码渲染文本的拼写信息。 The CLIP embedding may not encode spelling information of rendered text precisely.
该模型继承了欺骗性和有害内容的风险，且 AI 生成的痕迹更少。 The model inherits risks of deceptive and harmful content, with fewer traces of AI generation.
论文章节 · Sections（共 18）
摘要Abstract
1 引言1 Introduction
2 方法2 Method
2.1 解码器2.1 Decoder
2.2 先验2.2 Prior
3 图像操作3 Image Manipulations
3.1 变体3.1 Variations
3.2 插值3.2 Interpolations
3.3 文本差异3.3 Text Diffs
4 探索 CLIP 潜在空间4 Probing the CLIP Latent Space
5.1 先验的重要性5.1 Importance of the Prior
5.2 人工评估5.2 Human Evaluations
5.3 通过引导改进多样性-保真度权衡5.3 Improved Diversity-Fidelity Trade-off with Guidance