MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing

Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, Yinqiang Zheng

Abstract

Large-scale text-to-image (T2I) diffusion models have demonstrated remarkable capabilities in generating high-quality images. However, maintaining consistent object identity across different poses, viewpoints, or contexts remains a significant challenge. We present MasaCtrl, a tuning-free method that enables consistent image synthesis and complex non-rigid editing. By converting standard self-attention into Mutual Self-Attention, the framework allows a target generation process to query textures and identity features from a source image while adhering to a new target layout. This approach eliminates the need for time-consuming fine-tuning or optimization, providing an efficient solution for high-fidelity image manipulation.

1. Introduction

State-of-the-art diffusion models like Stable Diffusion excel at generating diverse imagery from textual prompts. Despite this, they often struggle with "identity drift"—where changing a prompt (e.g., from "a sitting dog" to "a running dog") results in a completely different animal. Achieving consistency is crucial for applications in storytelling, character design, and virtual reality.

Existing approaches to this problem generally fall into two categories: rigid editing and fine-tuning. Rigid editing methods, such as Prompt-to-Prompt, are effective for localized changes (e.g., color or style) but fail during non-rigid transformations like pose changes because they are constrained by the original spatial layout. Conversely, fine-tuning methods like Imagic can handle complex deformations but require several minutes of optimization per image, making them impractical for real-time applications. MasaCtrl bridges this gap by offering a tuning-free, inference-time modification that supports non-rigid editing while preserving identity.

2. Methodology

The fundamental insight behind MasaCtrl is that the self-attention layers within a U-Net architecture are responsible for maintaining local textures and object identity. By modifying these layers, we can control the synthesis process more effectively.

2.1 Mutual Self-Attention

In a standard diffusion process, self-attention allows the model to relate different spatial locations within the same image to ensure internal consistency. MasaCtrl introduces Mutual Self-Attention, where the "target" generation process (the edited image) is modified to query the Key ($K$) and Value ($V$) features from a "source" process (the original image). This allows the target image to "borrow" the specific textural details and identity markers of the source object while following the new structural guidance provided by the target prompt.

2.2 Strategic Timing and Placement

The authors observed that the global layout of an image is typically established in the early stages of the denoising process and within the shallower layers of the U-Net. To allow for non-rigid changes (like a change in pose), the model must first be allowed to establish the new layout. Consequently, MasaCtrl is strategically applied only to the decoder layers and only after a specific number of denoising steps. This decoupling ensures that the target image can adopt a new pose before the mutual attention mechanism begins "painting" the identity details from the source.

2.3 Mask-Guidance for Query Disambiguation

A common issue in mutual attention is "query confusion," where a foreground object might inadvertently query textures from the background of the source image. To mitigate this, MasaCtrl utilizes cross-attention maps to automatically generate masks. These masks guide the attention mechanism, ensuring that the foreground object in the target image only queries the corresponding foreground features in the source image.

3. Experimental Setup

As a tuning-free method, MasaCtrl operates during the inference phase and does not require a dedicated training dataset. The framework was evaluated using:

  • Stable Diffusion v1.4: As the primary base model.
  • Anything-V4: To demonstrate the method's robustness in the anime/illustration domain.
  • ControlNet & T2I-Adapter: To show compatibility with existing controllable generation frameworks.

4. Results and Discussion

MasaCtrl demonstrates superior performance in several key areas:

  • Non-Rigid Editing: The method successfully performs complex edits, such as altering the pose of an animal (e.g., a bird spreading its wings) while maintaining specific markings and textures that previous methods would lose.
  • Multi-View Consistency: By using a single "anchor" image as the source for mutual attention, MasaCtrl can generate a character from multiple camera angles (front, side, back) with high identity persistence.
  • Efficiency: Unlike optimization-based methods, MasaCtrl adds negligible computational overhead to the standard Stable Diffusion inference pipeline.
  • Integration: The method works seamlessly with ControlNet, allowing users to provide explicit structural guidance (like a skeleton or depth map) while MasaCtrl ensures the identity remains consistent with the source.

5. Limitations

Despite its effectiveness, MasaCtrl has limitations. It relies on the base model's ability to "hallucinate" content not present in the source image (e.g., the back of a character's head). If the required change is too extreme, the model may produce artifacts. Additionally, the method currently focuses on foreground consistency; in scenes with highly dynamic backgrounds, the mutual attention mechanism may struggle to decouple the moving subject from a static background.

6. Conclusion

MasaCtrl introduces a novel mechanism for identity-consistent image synthesis by hijacking the self-attention process. By enabling mutual querying between source and target images without the need for fine-tuning, it provides a versatile and efficient tool for the next generation of AI-driven image editing.

Summary updated

Abstract

Despite the success in large-scale text-to-image generation and text-conditioned image editing, existing methods still struggle to produce consistent generation and editing results. For example, generation approaches usually fail to synthesize multiple images of the same objects/characters but with different views or poses. Meanwhile, existing editing methods either fail to achieve effective complex non-rigid editing while maintaining the overall textures and identity, or require time-consuming fine-tuning to capture the image-specific appearance. In this paper, we develop MasaCtrl, a tuning-free method to achieve consistent image generation and complex non-rigid image editing simultaneously. Specifically, MasaCtrl converts existing self-attention in diffusion models into mutual self-attention, so that it can query correlated local contents and textures from source images for consistency. To further alleviate the query confusion between foreground and background, we propose a mask-guided mutual self-attention strategy, where the mask can be easily extracted from the cross-attention maps. Extensive experiments show that the proposed MasaCtrl can produce impressive results in both consistent image generation and complex non-rigid real image editing.
BibTeX
@inproceedings{cao2023masactrl,
  author = {Mingdeng Cao and Xintao Wang and Zhongang Qi and Ying Shan and Xiaohu Qie and Yinqiang Zheng},
  title = {{MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure