Abstract
Precise editing of real images using text-to-image diffusion models remains a significant challenge, often resulting in unwanted global changes or failing to handle non-rigid transformations. We present CPAM (Context-Preserving Adaptive Manipulation), a zero-shot framework designed for high-fidelity image editing without the need for model fine-tuning or optimization. CPAM introduces two key modules: Preservation Adaptation (PA) and Localized Extraction (LE), which surgically modify self-attention and cross-attention mechanisms to decouple foreground and background processing. Our approach enables complex edits, such as pose changes and object removals, while maintaining perfect background integrity. Experimental results on TEdBench and our new IMBA dataset demonstrate that CPAM outperforms existing zero-shot methods in both quantitative metrics and human preference.
Introduction
The advent of text-to-image diffusion models has revolutionized content creation, yet editing existing real-world photographs remains non-trivial. Most current editing techniques suffer from two primary issues. First, the "Global Change Trap" occurs when localized edits inadvertently alter the entire image's style or background. Second, the "Rigidity Trap" limits zero-shot methods to simple attribute changes (e.g., color), struggling with non-rigid transformations like changing a subject's pose. Furthermore, standard inversion techniques often lose fine-grained textures. CPAM addresses these limitations by providing a localized, non-rigid editing framework that preserves the original image's context.
Methodology
CPAM operates by modifying the internal attention mechanisms of Stable Diffusion during the denoising process. It consists of two primary components:
- Preservation Adaptation (PA) Module: This module targets self-attention. By utilizing masks, it independently controls foreground and background features. It extracts semantic content from the original image and injects it into the editing process, ensuring the background remains undistorted while allowing the foreground to adapt to new shapes or poses defined by the text prompt.
- Localized Extraction (LE) Module: This module focuses on cross-attention, where text prompts interact with image pixels. LE restricts pixels within the target mask to attend only to the edit prompt, while the rest of the image attends to "null text." This prevents the "leakage" of edit-related features into unwanted regions.
Additionally, a Mask-Guidance Strategy allows for flexible user input. Users can provide a simple source mask, which the system automatically refines into a "target mask" during diffusion to track objects as they change shape or position.
Experimental Evaluation
We evaluated CPAM using TEdBench and a newly introduced Image Manipulation BenchmArk (IMBA). IMBA provides a more comprehensive set of 104 samples with detailed user preferences and object-specific prompts.
Quantitative results show that CPAM achieves superior background preservation. In terms of LPIPS (Learned Perceptual Image Patch Similarity) for background regions, CPAM scored 0.149, significantly outperforming SDEdit (0.338) and MasaCtrl (0.223). Qualitative user studies further validated these findings, with CPAM being the preferred choice for both Object Retention (4.72/6) and Background Retention (5.09/6). Unlike many competitors, CPAM supports "Caption-Free" editing and "Object Removal" within a single unified framework.
Discussion and Limitations
CPAM demonstrates that sophisticated attention manipulation can achieve results previously thought to require fine-tuning (e.g., DreamBooth). However, as a zero-shot method, it is inherently limited by the base model's knowledge (e.g., SD-1.5). It may struggle with very small objects or regions not well-represented in the training data. Future work will explore integrating CPAM with more advanced base models like SDXL or Flux to further enhance generation quality.
Conclusion
CPAM provides a robust, training-free solution for complex real image editing. By decoupling foreground and background through targeted attention adaptation, it offers a level of precision and flexibility that bridges the gap between simple attribute editing and complex scene manipulation.
Summary updated