Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing

Bingyan Liu, Chengyu Wang, Tingfeng Cao, Kui Jia, Jun Huang

Abstract

Deep Text-to-Image Synthesis (TIS) models such as Stable Diffusion have recently gained significant popularity for creative text-to-image generation. However, for domain-specific scenarios, tuning-free Text-guided Image Editing (TIE) is of greater importance for application developers. This approach modifies objects or object properties in images by manipulating feature components in attention layers during the generation process. Nevertheless, little is known about the semantic meanings that these attention layers have learned and which parts of the attention maps contribute to the success of image editing. In this paper, we conduct an in-depth probing analysis and demonstrate that cross-attention maps in Stable Diffusion often contain object attribution information, which can result in editing failures. In contrast, self-attention maps play a crucial role in preserving the geometric and shape details of the source image during the transformation to the target image. Our analysis offers valuable insights into understanding cross and self-attention mechanisms in diffusion models. Furthermore, based on our findings, we propose a simplified, yet more stable and efficient, tuning-free procedure that modifies only the self-attention maps of specified attention layers during the denoising process. Experimental results show that our simplified method consistently surpasses the performance of popular approaches on multiple datasets.

Introduction

Text-guided Image Editing (TIE) aims to modify an existing image—such as changing a person to a robot—using only a text prompt. While models like Stable Diffusion excel at generating images from scratch, precise editing remains a challenge. Existing methods typically manipulate cross-attention (text-to-pixel) or self-attention (pixel-to-pixel) layers. However, these approaches can be unstable, often failing to preserve the original image's structure or background. This instability stems from a lack of understanding regarding the specific semantic information stored within these attention layers.

Probing the Attention Mechanisms

To bridge this gap, the authors conducted a probing analysis using a two-layer MLP classifier to predict object categories and colors from raw attention maps. The findings revealed two critical insights:

  1. Cross-Attention "Baggage": Cross-attention maps are not merely spatial weights; they carry rich semantic features. Swapping these maps can inadvertently inject source-image features into the target generation, leading to editing failures.
  2. Self-Attention as a "Skeleton": Self-attention maps capture the geometric and shape details (contours) of the source image. Preserving these maps allows the model to change texture or category while maintaining the original pose and position.

Free-Prompt-Editing (FPE)

Based on these insights, the authors introduced Free-Prompt-Editing (FPE), a simplified, tuning-free method. FPE operates through:

  • Self-Attention Hijacking: During the denoising of the edited image, the model "borrows" self-attention maps from the original image.
  • Layer Specificity: The replacement is restricted to layers 4 through 14, which the authors identified as the optimal range for balancing structural preservation with prompt-driven modification.
  • Prompt Flexibility: Unlike previous methods like Prompt-to-Prompt (P2P), FPE does not strictly require a source prompt, making it more practical for editing real-world photos where a perfect description may not exist.

Experimental Results

The authors evaluated FPE on several datasets, including Car-real-edit and ImageNet-real-edit, as well as standard benchmarks like Wild-TI2I.

Comparison of FPE (Ours) against P2P for complex edits like object transformation and color modification.

FPE consistently outperformed baselines in several key areas:

  • Structure Preservation: FPE achieved higher Clip Directional Similarity (CDS) scores, indicating better alignment with target prompts while maintaining the source layout.
  • Efficiency: FPE offers a massive speedup, editing an image in approximately 6.3 seconds on an NVIDIA A100—nearly 50x faster than the Plug-and-Play (PnP) method.
  • Qualitative Quality: The method successfully handles challenging tasks, such as transforming a person into a robot or changing a car's color without altering its specific model.

Conclusion and Limitations

FPE provides a robust and efficient framework for image editing by leveraging the structural properties of self-attention. However, it remains constrained by the generative limits of the base Stable Diffusion model. Future work will focus on improving the inversion process for high-fidelity facial editing and handling more complex scene-level rearrangements.

Summary updated

Abstract

Deep Text-to-Image Synthesis (TIS) models such as Stable Diffusion have recently gained significant popularity for creative text-to-image generation. However, for domain-specific scenarios, tuning-free Text-guided Image Editing (TIE) is of greater importance for application developers. This approach modifies objects or object properties in images by manipulating feature components in attention layers during the generation process. Nevertheless, little is known about the semantic meanings that these attention layers have learned and which parts of the attention maps contribute to the success of image editing. In this paper, we conduct an in-depth probing analysis and demonstrate that cross-attention maps in Stable Diffusion often contain object attribution information, which can result in editing failures. In contrast, self-attention maps play a crucial role in preserving the geometric and shape details of the source image during the transformation to the target image. Our analysis offers valuable insights into understanding cross and self-attention mechanisms in diffusion models. Furthermore, based on our findings, we propose a simplified, yet more stable and efficient, tuning-free procedure that modifies only the self-attention maps of specified attention layers during the denoising process. Experimental results show that our simplified method consistently surpasses the performance of popular approaches on multiple datasets. 11Source code and datasets are available at https://github.com/alibaba/EasyNLP/tree/master/diffusion/FreePromptEditing.
BibTeX
@misc{liu2024towards,
  author = {Bingyan Liu and Chengyu Wang and Tingfeng Cao and Kui Jia and Jun Huang},
  title = {{Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing}},
  year = {2024},
  eprint = {2403.03431},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.03431},
}
Figure