CLIP2GAN: Towards Bridging Text with the Latent Space of GANs

Yixuan Wang, Wengang Zhou, Jianmin Bao, Weilun Wang, Li Li

Abstract

In this work, we are dedicated to text-guided image generation and propose a novel framework, i.e., CLIP2GAN, by leveraging the CLIP model and StyleGAN. The key idea of our CLIP2GAN is to bridge the output feature embedding space of CLIP and the input latent space of StyleGAN, which is realized by introducing a mapping network. In the training stage, we encode an image with CLIP and map the output feature to a latent code, which is further used to reconstruct the image. In this way, the mapping network is optimized in a self-supervised learning way. In the inference stage, since CLIP can embed both image and text into a shared feature embedding space, we replace the CLIP image encoder in the training architecture with the CLIP text encoder, while keeping the following mapping network as well as the StyleGAN model. As a result, we can flexibly input a text description to generate an image. Moreover, by simply adding mapped text features of an attribute to a mapped CLIP image feature, we can effectively edit the attribute of the image. Extensive experiments demonstrate the superior performance of our proposed CLIP2GAN compared to previous methods.

1. Introduction

The field of AI-generated imagery has transitioned from stochastic generation to precise, text-driven synthesis. However, traditional text-to-image models often rely on massive datasets of paired images and captions, which are expensive to curate. Generative Adversarial Networks (GANs), particularly StyleGAN, provide high-fidelity outputs but lack intuitive control mechanisms, as their latent spaces do not naturally correspond to human language.

CLIP2GAN addresses this "alignment gap" by bridging CLIP (Contrastive Language-Image Pre-training) and StyleGAN. By introducing a mapping network that translates semantic embeddings into StyleGAN’s latent space, CLIP2GAN enables zero-shot text-to-image generation and fine-grained editing without requiring paired text-image data during training.

2. Methodology

The core of the CLIP2GAN framework is a 12-layer fully connected mapping network designed to align CLIP’s 512-dimensional feature space with StyleGAN’s $w+$ latent space.

2.1 Self-Supervised Training and Cross-Modal Inference

During the training phase, the model utilizes a self-supervised approach. An input image is processed by the CLIP image encoder to produce a semantic embedding. The mapping network then translates this embedding into a latent code, which StyleGAN uses to reconstruct the original image. Because CLIP is pre-trained to map both images and text into a shared multimodal embedding space, the mapping network implicitly learns to process text. During inference, the image encoder is simply replaced with the CLIP text encoder, allowing for direct text-to-image generation.

2.2 Adversarial Refinement and Diversity

CLIP embeddings primarily capture global semantics, which can lead to a loss of fine-grained structural details in the reconstruction. To mitigate this, the authors incorporate an external discriminator that trains adversarially with the mapping network, forcing it to recover high-frequency details like texture and hair. Furthermore, to ensure output diversity and prevent mode collapse (where the model generates the same "average" image for a given prompt), Gaussian noise is injected into the CLIP features, and a diversity loss (mode-seeking regularization) is applied.

3. Experimental Results

3.1 Datasets and Setup

The authors evaluated CLIP2GAN across several domains, including CelebA-HQ (30,000 celebrity faces), AFHQ (animal faces), and LSUN (churches and cars). While training remained text-free, the MM-CelebA-HQ dataset was used for testing text-alignment performance.

3.2 Quantitative and Qualitative Performance

CLIP2GAN demonstrates significant improvements over state-of-the-art methods such as TediGAN and StyleCLIP. On the CelebA-HQ benchmark, the model achieved a Frechet Inception Distance (FID) of 34.25, compared to StyleCLIP’s 101.75. It also achieved a superior LPIPS score of 0.408, indicating higher perceptual realism.

Sample of a high-fidelity face reconstruction generated by the CLIP2GAN framework.
Example demonstrating the diversity and realism of outputs produced by the mapping network.

Qualitative results show that CLIP2GAN excels at latent space arithmetic. By adding the mapped feature of a specific attribute (e.g., "smile") to an image's latent code, users can perform precise edits without the need for complex, per-image optimization.

4. Conclusion

CLIP2GAN successfully bridges the gap between semantic text understanding and high-fidelity image generation. By leveraging the multimodal properties of CLIP, it enables zero-shot generation and editing in a self-supervised manner. While some attribute entanglement remains a challenge for future work, the framework sets a new standard for efficient and flexible text-to-image synthesis.

Summary updated

Abstract

In this work, we are dedicated to text-guided image generation and propose a novel framework, i.e., CLIP2GAN, by leveraging CLIP model and StyleGAN. The key idea of our CLIP2GAN is to bridge the output feature embedding space of CLIP and the input latent space of StyleGAN, which is realized by introducing a mapping network. In the training stage, we encode an image with CLIP and map the output feature to a latent code, which is further used to reconstruct the image. In this way, the mapping network is optimized in a self-supervised learning way. In the inference stage, since CLIP can embed both image and text into a shared feature embedding space, we replace CLIP image encoder in the training architecture with CLIP text encoder, while keeping the following mapping network as well as StyleGAN model. As a result, we can flexibly input a text description to generate an image. Moreover, by simply adding mapped text features of an attribute to a mapped CLIP image feature, we can effectively edit the attribute to the image. Extensive experiments demonstrate the superior performance of our proposed CLIP2GAN compared to previous methods.
BibTeX
@article{wang2024clip,
  author = {Yixuan Wang and Wengang Zhou and Jianmin Bao and Weilun Wang and Li Li and Houqiang Li},
  title = {{CLIP2GAN: Towards Bridging Text with the Latent Space of GANs}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2024},
  url = {https://openreview.net/forum?id=jW9hmUisYl},
}
Figure