Abstract: The recent GAN inversion methods have been able to successfully invert the real image input to the corresponding editable latent code in StyleGAN. By combining with the language-vision model (CLIP), some text-driven image manipulation methods are proposed. However, these methods require extra costs to perform optimization for a certain image or a new attribute editing mode. To achieve a more efficient editing method, we propose a new Text-driven image Manipulation framework via Space Alignment (TMSA). The Space Alignment module aims to align the same semantic regions in CLIP and StyleGAN spaces. Then, the text input can be directly accessed into the StyleGAN space and be used to find the semantic shift according to the text description. The framework can support arbitrary image editing mode without additional cost. Our work provides the user with an interface to control the attributes of a given image according to text input and get the result in real time. Extensive experiments demonstrate our superior performance over prior works.
Introduction
Generative Adversarial Networks (GANs), particularly StyleGAN, have revolutionized high-fidelity image synthesis. However, manipulating existing real-world images remains a significant challenge. To edit a photo, it must first be "inverted" into a latent code within the GAN's space. Identifying the precise direction in this latent space that corresponds to a specific semantic change (e.g., adding a smile or changing hair color) typically requires either manual annotation of thousands of images, slow per-image optimization processes, or training specialized neural mappers for every single attribute.
This paper introduces TMSA (Text-driven image Manipulation via Space Alignment), a framework designed to bridge the gap between the semantic understanding of CLIP and the generative power of StyleGAN. TMSA provides a universal, efficient interface that translates arbitrary text prompts into precise latent space edits in real-time, maintaining high identity preservation without the need for extensive manual labeling or per-prompt optimization.
Methodology
The core of the TMSA framework is the Space Alignment module, which facilitates a direct mapping between CLIP's semantic embeddings and StyleGAN's $W+$ latent space. The methodology consists of three primary components:
1. Space Alignment Module
The researchers trained a Mapping Network to translate CLIP image embeddings into StyleGAN’s latent space. By leveraging CLIP’s joint language-vision property—where text and images sharing the same semantics are mapped to nearby points—the team realized that aligning the image space effectively aligns the text space as well. This allows the system to interpret text prompts as latent directions without requiring a dedicated text-to-latent dataset.
2. In-domain Adjustment
To ensure the mapping remains accurate and within the GAN's manifold, the authors utilized "synthetic" supervision. They sampled random latent codes from StyleGAN to generate images, which were then processed by CLIP. The Mapping Network was trained to reconstruct the original latent codes from these CLIP embeddings, ensuring the network stays "in-domain" and produces valid, high-quality edits.
3. Arbitrary Semantic Shift Extraction
To perform an edit, the system calculates a "semantic shift" ($\\Delta w$). It takes the CLIP embedding of a target text (e.g., "human face with glasses") and subtracts the embedding of a base text (e.g., "human face"). This difference is mapped into the StyleGAN space and added to the original image's latent code. This subtraction effectively cancels out common features, isolating the specific attribute change requested by the user.
Experimental Results
The authors evaluated TMSA on three standard high-resolution benchmarks: FFHQ (faces), LSUN Cars, and LSUN Church.
Qualitative Performance
As shown in the figure below, TMSA demonstrates remarkable versatility across multiple domains. It successfully handles abstract prompts (e.g., "Trump"), fine-grained facial attributes (e.g., "Mustache," "Sunglasses"), and structural changes in objects and architecture (e.g., "Jeep," "Baroque").
Quantitative Evaluation
- Identity Preservation: Using the ArcFace face recognition model, the authors confirmed that TMSA maintains the original subject's identity significantly better than previous methods like StyleCLIP or TediGAN.
- Accuracy: Using CLIP as a zero-shot classifier, the framework achieved 94-99% accuracy in successfully applying target attributes across most categories.
- User Preference: In a study with 35 participants, TMSA was preferred over 50% of the time compared to competitors, which often failed to reach 5% preference for complex attributes.
Conclusion
TMSA represents a significant step toward real-time, arbitrary text-driven image editing. By aligning the semantic spaces of CLIP and StyleGAN, the framework removes the need for expensive optimization and manual labeling. While the method is currently limited by the biases of its underlying pre-trained models, its "Space Alignment" concept offers a promising path for future research in other generative architectures, such as Diffusion Models.
Summary updated