Image editing with generative models presents a paradox: the more effectively a model follows a new instruction, the more likely it is to destroy the original image's identity. Unlike text-to-image generation, where the model has a blank canvas, editing requires a delicate balance between plasticity (the ability to change) and stability (the ability to preserve).
This tension is often described as the fidelity-editability trade-off. As Suho Ryu notes, increasing a model's responsiveness to new instructions often leads to a degradation in the preservation of unchanged regions. A user asking to "add sunglasses" to a portrait may find the model has also changed the subject's shirt color or blurred the background, a phenomenon Ashkan Mirzaei et al. describe as a lack of restraint in current tools.
The Source of the Conflict
The trade-off stems from how diffusion models process information. To edit a real image, it must first be "inverted" into the model's latent space. However, standard inversion techniques are often imperfect approximations. Jiahuan Wang et al. argue that these approximations degrade the unedited background, forcing a choice between high-quality edits that look out of place and realistic backgrounds that fail to reflect the requested changes.
Furthermore, the denoising process itself introduces accumulative drift. Chenxi Xie et al. explain that applying new text prompts during generation causes the model to deviate significantly from the original spatial layout. Without explicit constraints, the model treats the entire image as a target for change, leading to global artifacts or "hallucinations" in non-target areas, as observed by Rumeysa Bodur et al.
Navigating the Pareto Frontier
Researchers have proposed several mechanisms to navigate this "trilemma" of instruction following, context preservation, and seamless blending Weiyan Xie et al.
- Spatial and Temporal Guidance: Guoli Jia et al. suggest dynamically adjusting text guidance over time and space to handle "soft edits" like pose changes without losing the base model's creative priors. Similarly, Vadim Titov et al. use a self-guidance mechanism to balance requested modifications with the original lighting and structure.
- Control Loops and Memory: To prevent over-editing, Fengyi Fu et al. introduce a dynamic feedback loop that adaptively adjusts editing intensity at each denoising step. Haozhe Chi et al. treat editing as a control problem, using a Kalman filter to "remember" the original structure and maintain consistency.
- Latent Manipulation: Some methods attempt to isolate the edit to specific concepts. Jinqi Luo treats editing as a "compositional transplant," estimating the exact magnitude of latent vector shifts needed to swap concepts while leaving the rest of the image untouched. Qian Wang argues that manipulating the predicted noise ($\epsilon_t$) provides a superior balance for both local and global edits compared to manipulating attention maps alone.
Measuring the Balance
The subjectivity of what constitutes a "good" edit has led to a crisis in evaluation. Traditional metrics like CLIPScore often fail because they reward text alignment even if the background is destroyed.
To address this, Zhuoying Li et al. proposed the Balancing Preservation Modification (BPM) metric, which disentangles an image into editing-relevant and irrelevant regions to assess them independently. Similarly, Yiwei Ma et al. introduced I²EBench to evaluate models across 16 dimensions, revealing that no single architecture yet achieves universal proficiency in both high-level semantic changes and low-level restoration.
Ultimately, the goal of this research direction is to reach a Pareto frontier where edits are significant enough to be meaningful but restrained enough to be invisible. As Yeongtak Oh et al. suggest, the ideal system must balance the model's capacity for change against its ability to preserve the source's photorealism and identity.