Abstract
Although text-guided image manipulation approaches have demonstrated highly accurate performance for editing the appearance of images in a virtual or simple scenario, their real-world applications face significant challenges. The primary cause of these challenges is the misalignment in the distribution of training and real-world data, which leads to unstable text-guided image manipulation. In this work, we propose a novel framework called TolerantGAN and tackle the new task of real-world text-guided image manipulation independent of the training data. To achieve this, we introduce two key concepts of a border smoothly connection module (BSCM) and a manipulation direction-based attention module (MDAM). BSCM smoothens the misalignment in the distribution of training and real-world data. MDAM extracts only regions highly relevant for image manipulation and assists in reconstructing unobserved objects in the training data. For in-the-wild input images of various classes, TolerantGAN robustly outperforms the state-of-the-art methods.
Introduction
Most text-guided image manipulation tools, such as StyleCLIP, are trained on highly curated and perfectly aligned datasets (e.g., centered faces with clean backgrounds). However, when these tools are applied to "real-world" or "in-the-wild" photos—where subjects may be off-center, wearing unobserved accessories, or positioned in complex environments—the results often degrade significantly.
The authors identify this phenomenon as compositional misalignment. When a GAN attempts to reconstruct an "out-of-domain" image, it often fails to generate unobserved parts correctly, leading to artifacts. Furthermore, simply pasting a manipulated region back into the original photo creates jarring, visible seams. To address this, the paper formalizes the challenge of real-world images by defining "wildness" as the ratio of out-of-domain composition.
Methodology
The TolerantGAN architecture introduces two primary novel components designed to handle the complexities of real-world data while working alongside standard manipulators.
Border Smoothly Connection Module (BSCM)
Instead of a simple cut-and-paste operation, BSCM optimizes the latent code of the manipulated image so that its edges naturally "fade" into the original photo's margin. By utilizing a specialized border loss that combines Learned Perceptual Image Patch Similarity (LPIPS) and Mean Squared Error (MSE), the module ensures the generated image's boundary matches the real-world background while preserving the subject's identity.
Manipulation Direction-based Attention Module (MDAM)
MDAM identifies the specific regions where changes should occur. It leverages CLIP to compare the semantic "direction" of the text prompt (e.g., the shift from "no beard" to "beard") with the "direction" of the image features. This comparison generates an attention mask that isolates only the regions relevant to the text prompt. This selective fusion prevents the GAN from attempting to reconstruct (and potentially ruining) parts of the image it does not understand, such as complex backgrounds or accessories.
Key Results
TolerantGAN was evaluated across diverse datasets, including FFHQ for faces, Oxford-IIIT PET for animals, and LSUN for architectural structures.
- Quantitative Performance: On the FFHQ dataset, TolerantGAN achieved a higher CLIP score (0.39) and SSIM (0.96) compared to baselines, indicating superior alignment with text prompts and better structural consistency. It also demonstrated a lower FID (19.19), signifying higher overall image quality.
- Qualitative Performance: Visual comparisons show that while standard models create cartoonish artifacts or fail to handle "wild" attributes like unaligned poses, TolerantGAN produces seamless, photo-realistic edits that maintain the integrity of the original frame.
Conclusion and Future Work
TolerantGAN successfully bridges the gap between curated training environments and real-world application. However, the framework currently faces limitations regarding inference speed; the BSCM optimization process requires approximately 100 seconds per image on an NVIDIA V100 GPU. Future research will explore the application of these "tolerance" concepts to diffusion models and the implementation of encoder-based methods to achieve real-time performance.
Summary updated