TL;DR
ConserveEdit is an editor-agnostic controller that prevents "drift"—unintended changes to backgrounds or protected objects—during iterative image editing. By treating each refinement step as a transaction against a "contract" anchored to the original source image, it ensures that pixels outside the edit region remain bit-identical to the original while strictly auditing for edit progress and image quality.
What Problem Does This Paper Address?
Instruction-driven image editors (like InstructPix2Pix or Qwen-Image-Edit) are impressive, but they struggle with iterative refinement. Because these models typically re-render the entire scene for every edit, small unintended changes accumulate over multiple turns. This is known as drift.
Even when using masks, standard editors often cause global color shifts or distort nearby sensitive content like faces and text. Simple masking isn't a silver bullet because automatic grounding is imperfect; if a mask slightly overlaps a person's face, a "make the train smile" instruction might accidentally hallucinate features onto that face. The authors argue that current "agentic" editors fail because they rely on aggregate quality scores that allow a strong edit to "compensate" for a corrupted background.
Methods
ConserveEdit introduces a transactional refinement loop that treats the original source image ($x_0$) as the immutable source of truth.
- The Edit Contract: Before editing begins, a contract is compiled from the source image and the instruction. It defines the authorized support (the "core" mask of what should change plus a "halo" for shadows/reflections) and protected entities (background, faces, and text) with explicit tolerances.
- Source-Anchored Masking: Every candidate generated by a frozen editor is composed with the source image through a mask. This ensures that every pixel outside the authorized region is bit-exact to the original $x_0$.
- Multi-Dimensional Audit: Instead of one aggregate score, ConserveEdit subjects each candidate to three independent tests:
- Progress: Does the edit actually follow the instruction?
- Drift: Have protected entities (faces, text) changed beyond a calibrated threshold relative to $x_0$?
- Quality: Are there visible seams or artifacts at the mask boundaries?
- Rollback and Repair: If no candidate passes all tests, the system rolls back the transaction and attempts a "repair," such as shrinking the mask to avoid a protected entity or expanding the halo to capture a missing shadow.
Datasets
- MagicBrush: The primary real-world benchmark, consisting of 1,053 multi-turn image editing sequences with human-drawn masks and ground-truth targets.
- SimEdit: A new procedural simulator introduced by the authors. It generates scenes with known ground-truth masks for targets, shadows, faces, and text, allowing for precise measurement of drift that is impossible with real-world data where the "true" causal support of an edit is unknown.
Key Insights and Ideas
- Goalpost Drift: Most iterative editors compare the current step to the previous step. ConserveEdit always compares back to the original source. This prevents the "telephone game" effect where small errors compound into total image degradation.
- Masking is Necessary but Not Sufficient: As shown in Figure 1, even with a mask, direct editing often fails to protect nearby sensitive content. Masking provides bit-exactness for the background, but the audit is what protects the content inside or near the mask.
- Non-Substitutability: The authors prove that no single scalar weight (e.g., $EditScore - \lambda \cdot Drift$) can replicate their results. You cannot "buy" a better edit by sacrificing the integrity of a human face.
Key Results
- Bit-Exact Preservation: ConserveEdit guarantees 0% drift for all pixels outside the authorized region.
- Face/Text Protection: On MagicBrush turns involving automatic masks, standard masked editing violated protected faces or text 48% of the time. ConserveEdit reduced this to just 2%.
- Constrained Success (CS): On the same task, ConserveEdit achieved a CS of 53%, more than double the 23% achieved by standard scalar selection methods.
- DINO Similarity: Using a frozen Qwen-Image-Edit, ConserveEdit raised the DINO similarity to ground truth from 0.67 (direct) to 0.85, matching the performance of human-masked baselines but with automated protection.
Limitations and Future Work
- Grounding Bottleneck: The system relies heavily on the initial grounding (e.g., Grounding DINO + SAM). If the initial mask misses the target entirely, the edit will fail or be incomplete.
- Boundary Seams: While the quality audit helps, re-rendering editors often produce systematic seams at mask boundaries. Future work could integrate better harmonization or diffusion-based blending within the halo.
- Predicate Noise: The local CLIP-based progress scores can be noisy, sometimes leading to "false rejections" where a valid edit is discarded because the score didn't hit a specific threshold.
- Computational Cost: Generating multiple candidates and running multiple audits per turn is significantly more expensive than a single "direct" edit call.
Summary updated