Shifted Diffusion for Text-to-image Generation

Bingchen Liu, Yufan Zhou, Yizhe Zhu, Changyou Chen, Jinhui XuOpenReview 2022 · DOI · OpenReview

Abstract

We present Corgi, a novel method for text-to-image generation. Corgi is based on our proposed shifted diffusion model, which achieves better image embedding generation from input text. Unlike the baseline diffusion model used in DALL-E 2, our method seamlessly encodes prior knowledge of the pre-trained CLIP model in its diffusion process by designing a new initialization distribution and a new transition step of the diffusion. Compared to the strong DALL-E 2 baseline, our method performs better in generating image embedding from the text in terms of both efficiency and effectiveness, resulting in better text-to-image generation. Extensive large-scale experiments are conducted and evaluated in terms of both quantitative measures and human evaluation, indicating a stronger generation ability of our method compared to existing ones. Furthermore, our model enables semi-supervised and language-free training for text-to-image generation, where only part or none of the images in the training dataset have an associated caption. Trained with only 1.7% of the images being captioned, our semi-supervised model obtains FID results comparable to DALL-E 2 on zero-shot text-to-image generation evaluated on MS-COCO. Corgi also achieves new state-of-the-art results across different datasets on downstream language-free text-to-image generation tasks, outperforming the previous method, Lafite, by a large margin.

Introduction

Modern text-to-image (T2I) systems, such as DALL-E 2, typically employ a two-stage process: a "prior" model translates a text prompt into a visual representation (an image embedding), and a decoder then generates the final image from that embedding. However, this approach faces significant challenges. First, the "modality gap" in CLIP-like models means that image and text embeddings occupy different regions of the vector space, making alignment difficult. Second, standard diffusion models initialize from pure Gaussian noise, which is inefficient because valid image embeddings occupy only a tiny fraction of the total embedding space.

To address these issues, we introduce Corgi (Clip-based shifted diffusiOn model bRidGIng the gap). Corgi leverages a novel Shifted Diffusion process that starts from a distribution closer to the target CLIP embedding space, significantly improving image-text alignment and generation quality.

Methodology: Shifted Diffusion

The core innovation of Corgi is the redesign of the diffusion process. Instead of starting from standard Gaussian noise ($N(0, I)$), Corgi utilizes a "shifted" distribution that incorporates prior knowledge of the image embedding space.

Extracted figure from page 2 of Shifted Diffusion for Text-to-image Generation

Key Components

  • Shifted Initialization: The diffusion process begins with a distribution $N(\mu, \Sigma)$ derived from the actual distribution of image embeddings in the training set. This ensures the starting point is semantically relevant.
  • Multi-modal Prior: To account for the diversity of visual concepts, we model the image embedding space as a collection of $k$ Gaussians. The model selects the most relevant cluster based on the text prompt for initialization.
  • Flexible Training Regimes: Corgi supports supervised, semi-supervised (using as little as 1.7% captioned data), and language-free training modes.

Experimental Setup

We trained Corgi on a massive dataset of 900 million image-text pairs. For evaluation, we used MS-COCO for zero-shot FID scores and domain-specific datasets like CUB and MM-CelebA-HQ for language-free tasks. Qualitative assessments were conducted using DrawBench and PartiPrompts.

Results and Discussion

Corgi demonstrates superior performance across both quantitative and qualitative metrics. On MS-COCO, Corgi achieved an FID of 10.88. Remarkably, the semi-supervised variant (Corgi-Semi), trained with only 1.7% of captions, achieved an FID of 10.60, rivaling fully supervised state-of-the-art models.

Extracted figure from page 2 of Shifted Diffusion for Text-to-image Generation

In language-free generation, Corgi significantly outperformed previous methods like Lafite. Human evaluations on DrawBench and PartiPrompts also showed a clear preference for Corgi over DALL-E 2 in terms of both image fidelity and prompt alignment. These results suggest that by improving the "prior" model through shifted diffusion, we can achieve higher quality generation with significantly less labeled data.

Conclusion and Future Work

Corgi demonstrates that the initialization point of the diffusion process is critical for efficient and effective text-to-image generation. By "shifting" the diffusion toward the valid image embedding manifold, we bridge the modality gap inherent in CLIP-based systems. While Corgi is currently dependent on the quality of the underlying CLIP model, future work will explore applying shifted diffusion to other modalities like audio and video, as well as investigating more complex initialization distributions.

BibTeX
@misc{liu2022shifted,
  author = {Bingchen Liu and Yufan Zhou and Yizhe Zhu and Changyou Chen and Jinhui Xu},
  title = {{Shifted Diffusion for Text-to-image Generation}},
  year = {2022},
  eprint = {2211.15388},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.15388},
  howpublished = {OpenReview},
}
Figure