CLIP2GAN: Toward Bridging Text With the Latent Space of GANs

Yixuan Wang, Wengang Zhou, Jianmin Bao, Weilun Wang, Li Li

CLIP2GAN: Bridging the Gap Between Words and Pixels

In the world of AI-generated art, we’ve seen a massive shift from "randomly generated faces" to "images generated from precise text prompts." However, training these models usually requires gargantuan datasets of paired images and captions. CLIP2GAN changes the game by bridging two of the most powerful tools in modern AI—CLIP and StyleGAN—to enable high-quality text-to-image generation and editing without needing a single caption during training.

TL;DR

CLIP2GAN introduces a mapping network that translates CLIP’s semantic embeddings into StyleGAN’s latent space, enabling zero-shot text-to-image generation. By training on images alone, the model learns to generate diverse, high-fidelity images and perform fine-grained editing through simple latent space arithmetic.

What Problem Does This Paper Address?

Generative Adversarial Networks (GANs), specifically StyleGAN, are the gold standard for creating hyper-realistic images. However, controlling them is notoriously difficult; you usually have to tweak abstract "latent codes" that don't correspond to human language.

On the other hand, CLIP (Contrastive Language-Image Pre-training) is a master at understanding the relationship between text and images. While researchers have tried to combine them before, previous methods often required expensive paired text-image datasets or slow, per-image optimization (like StyleCLIP). CLIP2GAN addresses the "alignment gap"—the fact that CLIP’s feature space and StyleGAN’s latent space don't naturally speak the same language—allowing for fast, flexible, and "text-free" training.

Methods

The core of CLIP2GAN is a 12-layer fully connected mapping network. Here is how the authors bridge the two models:

  1. Self-Supervised Training: During training, the model only sees images. An image is fed into the CLIP image encoder. The mapping network then translates that CLIP embedding into a latent code ($w+$) for StyleGAN, which reconstructs the original image.
  2. The "Text-Free" Secret: Because CLIP is pre-trained to embed images and text into the same shared space, the mapping network "accidentally" learns how to handle text. During inference, you simply swap the CLIP image encoder for the CLIP text encoder.
  3. Adversarial Refinement: CLIP embeddings are 512-dimensional and tend to capture global semantics while losing fine details (like skin texture or individual hairs). To fix this, the authors add a discriminator that trains adversarially with the mapping network to "fill in" the missing details, ensuring the output is crisp and realistic.
  4. Diversity via Noise: To prevent the model from outputting the same "average" face for a prompt, they inject Gaussian noise into the CLIP features and use a diversity loss (mode-seeking regularization) to ensure different noise inputs result in distinct, high-quality images.

Datasets

The authors evaluated CLIP2GAN across several diverse domains:

  • CelebA-HQ: 30,000 high-resolution celebrity faces.
  • MM-CelebA-HQ: A version of the face dataset with 10 unique text descriptions per image (used for testing, not training).
  • AFHQ (Animal Faces): High-quality images of cats and dogs.
  • LSUN: Specifically the "Church" and "Car" categories to demonstrate generalization beyond faces.

Key Insights and Ideas

  • Zero-Shot is Possible: The most striking insight is that you don't need captions to build a text-to-image model. By leveraging the "multimodal" nature of CLIP, the model inherits text-understanding for free.
  • Latent Arithmetic for Editing: Because the model maps text directly into StyleGAN’s structured latent space, editing an image becomes a simple math problem. To make a person smile, you take the mapped feature for the word "smile" ($n$) and add it to the image's latent code ($z$): $z' = z + \beta n$. No complex optimization required.
  • Bridging the Detail Gap: The authors recognized that CLIP is a "semantic" encoder, not a "structural" one. Their use of an external discriminator to recover the "lost" 512-dimensional details is a clever architectural fix for the blurriness often seen in CLIP-based reconstructions.

Key Results

CLIP2GAN significantly outperformed existing state-of-the-art methods like AttnGAN, TediGAN, and StyleCLIP:

  • Quantitative: On the CelebA-HQ dataset, CLIP2GAN achieved a FID (Frechet Inception Distance) of 34.25, a massive improvement over StyleCLIP (101.75) and TediGAN (106.37). A lower FID indicates higher image quality and diversity.
  • Perceptual Quality: It also led in LPIPS (0.408), meaning the generated images were more perceptually similar to real human faces.
  • User Preference: In a paired user study, participants preferred CLIP2GAN’s results over 80% of the time compared to older models, citing better alignment with the text prompts and higher realism.

Limitations and Future Work

  • Attribute Entanglement: While the model is excellent at editing, some attributes are still "entangled." For example, asking the model to make a face "older" might inadvertently change the person's gender or hair color if those features are correlated in the training data.
  • StyleGAN Constraints: The model can only generate what the underlying StyleGAN has seen. If you use a StyleGAN trained on faces, CLIP2GAN cannot generate a "cat" even if CLIP understands the word.
  • Future Directions: The authors suggest exploring more complex mapping architectures and investigating ways to further disentangle attributes in the latent space to allow for even more precise "surgical" edits to images.

Summary updated

Abstract

In this work, we are dedicated to text-guided image generation and propose a novel framework, i.e., CLIP2GAN, by leveraging CLIP model and StyleGAN. The key idea of our CLIP2GAN is to bridge the output feature embedding space of CLIP and the input latent space of StyleGAN, which is realized by introducing a mapping network. In the training stage, we encode an image with CLIP and map the output feature to a latent code, which is further used to reconstruct the image. In this way, the mapping network is optimized in a self-supervised learning way. In the inference stage, since CLIP can embed both image and text into a shared feature embedding space, we replace CLIP image encoder in the training architecture with CLIP text encoder, while keeping the following mapping network as well as StyleGAN model. As a result, we can flexibly input a text description to generate an image. Moreover, by simply adding mapped text features of an attribute to a mapped CLIP image feature, we can effectively edit the attribute to the image. Extensive experiments demonstrate the superior performance of our proposed CLIP2GAN compared to previous methods.
BibTeX
@article{wang2024clip,
  author = {Yixuan Wang and Wengang Zhou and Jianmin Bao and Weilun Wang and Li Li and Houqiang Li},
  title = {{CLIP2GAN: Toward Bridging Text With the Latent Space of GANs}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2024},
  url = {https://openreview.net/forum?id=jW9hmUisYl},
}

Loading related content...

Figure