CPAM: Context-Preserving Adaptive Manipulation for Zero-Shot Real Image Editing

Dinh-Khoi Vo, Thanh-Toan Do, Tam Van Nguyen, Minh-Triet Tran, Trung-Nghia LearXiv 2025 · DOI · arXiv

Abstract

Precise editing of real images using text-to-image diffusion models remains a significant challenge, often resulting in unwanted global changes or failing to handle non-rigid transformations. We present CPAM (Context-Preserving Adaptive Manipulation), a zero-shot framework designed for high-fidelity image editing without the need for model fine-tuning or optimization. CPAM introduces two key modules: Preservation Adaptation (PA) and Localized Extraction (LE), which surgically modify self-attention and cross-attention mechanisms to decouple foreground and background processing. Our approach enables complex edits, such as pose changes and object removals, while maintaining perfect background integrity. Experimental results on TEdBench and our new IMBA dataset demonstrate that CPAM outperforms existing zero-shot methods in both quantitative metrics and human preference.

Introduction

The advent of text-to-image diffusion models has revolutionized content creation, yet editing existing real-world photographs remains non-trivial. Most current editing techniques suffer from two primary issues. First, the "Global Change Trap" occurs when localized edits inadvertently alter the entire image's style or background. Second, the "Rigidity Trap" limits zero-shot methods to simple attribute changes (e.g., color), struggling with non-rigid transformations like changing a subject's pose. Furthermore, standard inversion techniques often lose fine-grained textures. CPAM addresses these limitations by providing a localized, non-rigid editing framework that preserves the original image's context.

Figure 1: CPAM demonstrates versatile editing capabilities, including object replacement, removal, and complex pose changes while preserving the background.

Methodology

CPAM operates by modifying the internal attention mechanisms of Stable Diffusion during the denoising process. It consists of two primary components:

  • Preservation Adaptation (PA) Module: This module targets self-attention. By utilizing masks, it independently controls foreground and background features. It extracts semantic content from the original image and injects it into the editing process, ensuring the background remains undistorted while allowing the foreground to adapt to new shapes or poses defined by the text prompt.
  • Localized Extraction (LE) Module: This module focuses on cross-attention, where text prompts interact with image pixels. LE restricts pixels within the target mask to attend only to the edit prompt, while the rest of the image attends to "null text." This prevents the "leakage" of edit-related features into unwanted regions.

Additionally, a Mask-Guidance Strategy allows for flexible user input. Users can provide a simple source mask, which the system automatically refines into a "target mask" during diffusion to track objects as they change shape or position.

Figure 2: Example of background alteration using CPAM. The red car is preserved while the background is successfully transitioned to a Manhattan street scene.

Experimental Evaluation

We evaluated CPAM using TEdBench and a newly introduced Image Manipulation BenchmArk (IMBA). IMBA provides a more comprehensive set of 104 samples with detailed user preferences and object-specific prompts.

Quantitative results show that CPAM achieves superior background preservation. In terms of LPIPS (Learned Perceptual Image Patch Similarity) for background regions, CPAM scored 0.149, significantly outperforming SDEdit (0.338) and MasaCtrl (0.223). Qualitative user studies further validated these findings, with CPAM being the preferred choice for both Object Retention (4.72/6) and Background Retention (5.09/6). Unlike many competitors, CPAM supports "Caption-Free" editing and "Object Removal" within a single unified framework.

Discussion and Limitations

CPAM demonstrates that sophisticated attention manipulation can achieve results previously thought to require fine-tuning (e.g., DreamBooth). However, as a zero-shot method, it is inherently limited by the base model's knowledge (e.g., SD-1.5). It may struggle with very small objects or regions not well-represented in the training data. Future work will explore integrating CPAM with more advanced base models like SDXL or Flux to further enhance generation quality.

Conclusion

CPAM provides a robust, training-free solution for complex real image editing. By decoupling foreground and background through targeted attention adaptation, it offers a level of precision and flexibility that bridges the gap between simple attribute editing and complex scene manipulation.

Summary updated

Abstract

Editing natural images using textual descriptions in text-to-image diffusion models remains a significant challenge, particularly in achieving consistent generation and handling complex, non-rigid objects. Existing methods often struggle to preserve textures and identity, require extensive fine-tuning, and exhibit limitations in editing specific spatial regions or objects while retaining background details. This paper proposes Context-Preserving Adaptive Manipulation (CPAM), a novel zero-shot framework for complicated, non-rigid real image editing. Specifically, we propose a preservation adaptation module that adjusts self-attention mechanisms to preserve and independently control the object and background effectively. This ensures that the objects' shapes, textures, and identities are maintained while keeping the background undistorted during the editing process using the mask guidance technique. Additionally, we develop a localized extraction module to mitigate the interference with the non-desired modified regions during conditioning in cross-attention mechanisms. We also introduce various mask-guidance strategies to facilitate diverse image manipulation tasks in a simple manner. CPAM can be seamlessly integrated with multiple diffusion backbones, including SD1.5, SD2.1, and SDXL, demonstrating strong generalization across different model architectures. Extensive experiments on our newly constructed Image Manipulation BenchmArk (IMBA), a robust benchmark dataset specifically designed for real image editing, demonstrate that our proposed method is the preferred choice among human raters, outperforming existing state-of-the-art editing techniques. The source code and data will be publicly released at the project page: https://vdkhoi20.github.io/CPAM
BibTeX
@misc{vo2025cpam,
  author = {Dinh-Khoi Vo and Thanh-Toan Do and Tam Van Nguyen and Minh-Triet Tran and Trung-Nghia Le},
  title = {{CPAM: Context-Preserving Adaptive Manipulation for Zero-Shot Real Image Editing}},
  year = {2025},
  eprint = {2506.18438},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.18438},
}
Figure