Preserving Background and Structure in Local Image Editing

592 concepts432 papers2 proposals

When you ask a modern generative model to edit an image—such as replacing a dog with a cat—you might expect the model to alter only the pixels belonging to the animal. In practice, however, deep generative models often struggle with localization. Changing a single word in a text prompt frequently triggers a complete regeneration of the scene, causing unwanted shifts in lighting, background textures, and overall composition.

This phenomenon, known as spatial editing leakage or unintended global modification, stems from a fundamental tension: generative models are trained to optimize global image coherence, making them inherently resistant to isolated, surgical modifications.

Why Spatial Leakage Occurs

To understand why backgrounds drift and structures distort during editing, we have to look at how text-to-image models process information.

1. The Global Denoising Space

In diffusion models, image generation occurs by iteratively removing noise from a full-frame latent tensor. When an edit instruction is introduced, it alters the text conditioning vector applied to the entire image. As highlighted by Gu et al. in their work on PHOTOSWAP, without explicit spatial constraints, this global shift in context forces the model to recalculate features for unedited regions, leading to a loss of the original background and composition.

2. Background Drift and Feature Bleeding

During sequential or iterative editing, this problem compounds. Kim et al. demonstrate that standard diffusion models suffer from "background drift" because they lack a mechanism to lock down the exact state of unedited areas across multiple generation cycles. Furthermore, the cross-attention mechanisms that map text tokens to spatial regions are often blurry. As a result, instructions meant for a specific object bleed into surrounding pixels, causing global color shifts or structural distortions, a challenge documented in Instruct-CLIP.

[Text Prompt: "Change dog to cat"] 
       │
       ▼
[Global Cross-Attention] ──► (Blurs boundaries) ──► [Unintended Changes to Background]

Mechanisms for Surgical Control

To prevent global leakage, researchers have developed several distinct strategies to anchor unedited regions while allowing the target zone to transform freely.

Restricting the Generation Space

One intuitive way to prevent a model from altering the background is to limit its capacity to do so. Chen et al. introduce this concept in PaRa by explicitly restricting the model's learnable parameter space using rank reduction. By narrowing the degrees of freedom available during fine-tuning, the model is forced to find solutions that reuse the existing image structure rather than hallucinating an entirely new background.

Attention and Feature Anchoring

Because attention maps dictate where information is distributed, manipulating them provides a direct lever for spatial control.

  • In Diffusion Models: Gu et al. manipulate self- and cross-attention layers to ensure that the layout, geometry, and background from the source image are directly injected into the generated target image.
  • In GAN Architectures: Li et al. address this in ManiGAN via an Affine Combination Module (ACM). Instead of concatenating text and image features globally, the ACM surgically selects which image features to modify based on the text prompt, keeping irrelevant background features intact.

Localized Token Prediction

Instead of feeding the entire image through a denoising pipeline and trying to hold the background constant, why not only generate the pixels that need to change?

Wu et al. pioneer this approach with NEP (Next Editing Token Prediction). By framing image editing as an autoregressive token-prediction task, NEP restricts the model to predicting tokens exclusively within a user-specified editing mask. Because the pixels outside the mask are never processed or regenerated by the model, background degradation is mathematically impossible.

Similarly, Tang et al. introduce SPICE, which pairs localized "context dots" with structural control networks to allow users to execute over 100 consecutive edits without experiencing background degradation.

Structural Priors for Global Style Changes

The challenge of structural preservation extends to global transformations like artistic style transfer, where the goal is to change texture and color without altering geometry. Zhang et al. demonstrate in DyArtbank that large-scale models tend to hallucinate new features during stylization, distorting human faces or architecture. To combat this, Zhang et al. introduce a Style Prior loss in SPAST that acts as a structural anchor, ensuring that the original semantic layout remains fixed even under intense stylistic distortion.

Object-Centric Slot Decomposition

Another emerging paradigm bypasses text entirely to isolate edits. Kaan Akan introduces SlotAdapt, which integrates slot attention into pre-trained diffusion models. By decomposing an image into discrete, independent object "slots," a user can add, remove, or modify a specific object slot without any risk of spatial leakage into the surrounding background slots.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_unintended_global_modification_bc1e03a0,
  title = {unintended global modification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unintended-global-modification/rdr_bc1e03a0a28a4fdd8e0e250404c49014},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_unintended_global_modification_bc1e03a0_v1,
  title = {unintended global modification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unintended-global-modification/rdr_bc1e03a0a28a4fdd8e0e250404c49014/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake