Text Guided Image Editing with Automatic Concept Locating and Forgetting

Jia Li, Lijie Hu, Zhixian He, Jingfeng Zhang, Tianhang Zheng, Di WangarXiv 2024 · DOI · arXiv

Abstract

Text-to-image diffusion models often struggle with image editing due to the "ghosting" effect, where attributes of the original object persist in the edited output. We present the Locate and Forget (LaF) framework, which automates the editing process by identifying conflicting concepts between the original image and the target prompt. By applying negative guidance to "unlearn" these concepts during diffusion, LaF achieves high-fidelity edits without manual masking. Evaluations on TedBench and MagicBrush demonstrate LaF's superiority in semantic alignment and image quality over existing baselines.

Introduction

The primary challenge in text-guided image editing is the semantic gap between text instructions and pixel-level modifications. When prompted to change a "red car" into a "yellow bus," standard diffusion models often produce a hybrid result—a yellow vehicle that retains the car's silhouette. This "stubbornness" stems from the model's inability to reconcile the original visual structure with the new textual intent. While manual masking can mitigate this, it is labor-intensive and non-scalable. LaF addresses this by automatically locating what needs to be changed and ensuring those features are forgotten during the generation process.

Methodology

The LaF pipeline consists of two main stages:

Automatic Concept Locating

The system utilizes a vision-language model (OpenFlamingo) to generate a detailed scene description of the input image. It then performs dependency parsing on both the scene description and the editing prompt. By comparing syntactic trees, the model identifies specific conflicting attributes (e.g., "red" and "car" vs. "yellow" and "bus"). This allows for precise identification of subjects and their modifiers, which is more effective than simple keyword matching.

Original input image featuring a red car.

Concept Forgetting via Negative Guidance

Rather than simply overwriting the image, LaF modifies the Classifier-Free Guidance (CFG) equation. It calculates the noise associated with the "to-be-forgotten" concepts and subtracts it from the denoising process. This active subtraction clears the latent space for the new prompt's attributes, preventing the "ghosting" effect where old shapes or colors linger. The intensity of this process is controlled by a forgetfulness scale ($\eta$), which balances concept removal with overall image coherence.

Experimental Evaluation

LaF was evaluated across three benchmarks to ensure robustness:

  • TedBench: A public benchmark of 100 specific image editing tasks.
  • MagicBrush: A large-scale synthetic dataset containing over 500 complex real-world scenarios.
  • Human Prompt Generated Dataset: A set of 700 image-text triplets where the base images were generated by Stable Diffusion and then edited based on human-written instructions.

Results

LaF consistently outperformed state-of-the-art baselines like HIVE and InstructPix2Pix (IP2P). On TedBench, LaF achieved a CLIP-T score of 28.94 (vs. SD's 28.35) and an Inception Score of 18.69 (vs. IP2P's 15.81). Qualitatively, LaF successfully performs complex edits, such as completely changing object categories or removing subjects, without leaving artifacts. In a user study with over 100 participants, LaF was rated as the preferable method more often than competing models.

Edited output using LaF: a yellow bus successfully replaces the red car without ghosting artifacts.

Conclusion

LaF provides a robust, mask-free solution for text-guided image editing. By combining NLP-driven scene analysis with negative guidance, it effectively solves the ghosting problem. While the model currently faces limitations in numerical precision and complex spatial reasoning, it represents a significant step toward automated, high-fidelity image manipulation. Future work will focus on integrating advanced grounding modules to further refine the "Locate" phase of the framework.

Summary updated

Abstract

With the advancement of image-to-image diffusion models guided by text, significant progress has been made in image editing. However, a persistent challenge remains in seamlessly incorporating objects into images based on textual instructions, without relying on extra user-provided guidance. Text and images are inherently distinct modalities, bringing out difficulties in fully capturing the semantic intent conveyed through language and accurately translating that into the desired visual modifications. Therefore, text-guided image editing models often produce generations with residual object attributes that do not fully align with human expectations. To address this challenge, the models should comprehend the image content effectively away from a disconnect between the provided textual editing prompts and the actual modifications made to the image. In our paper, we propose a novel method called Locate and Forget (LaF), which effectively locates potential target concepts in the image for modification by comparing the syntactic trees of the target prompt and scene descriptions in the input image, intending to forget their existence clues in the generated image. Compared to the baselines, our method demonstrates its superiority in text-guided image editing tasks both qualitatively and quantitatively.
BibTeX
@misc{li2024text,
  author = {Jia Li and Lijie Hu and Zhixian He and Jingfeng Zhang and Tianhang Zheng and Di Wang},
  title = {{Text Guided Image Editing with Automatic Concept Locating and Forgetting}},
  year = {2024},
  eprint = {2405.19708},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.19708},
}
Figure