Instruction-based image editors often change parts of the picture the user never asked about. Ask for "make the dog a cat" and the grass, the lighting, or a second animal may shift too. We propose that much of this leakage comes from a reasoning failure rather than a rendering failure. Consider an instruction that names its target only indirectly, such as "recolor the mug held by the person standing left of the window." The editor must first work out which object is meant, a chain of lookups similar to multi-hop question answering. When it cannot work that out, it spreads the edit across every plausible candidate. We predict that leakage grows with the number of reference hops and falls mostly on "distractors," the objects that could have been the target but were not.
The fix we propose is a data one. We generate multi-hop edit instructions from explicit scene programs, so the true target region is known exactly by construction. We then keep only synthetic edits whose pixel changes stay inside that region, and distill the surviving data into a single-pass editor. The main novelty is that the program supplies an exact, per-hop ground-truth mask for filtering. Today's editing-data filters rely on vision-language judges or on masks those judges annotate, and both are known to miss subtle background changes.
Core Claim
We make two linked claims. The diagnostic claim is that on current editors, the amount of change outside the target (measured as PSNR, a pixel-fidelity score, restricted to unedited regions) drops as the number of reference hops rises, and that this change concentrates on same-category distractors. The method claim is that training on edits filtered with program-derived masks reduces this distractor leakage. It should do better than the same data filtered by a learned reward model and better than a pipeline that first rewrites the instruction into a direct one. It should also keep the lighting and shadow coherence that hard-mask methods lose.
Grounding in Prior Work
Three editing papers motivate the gap. CoCoEdit runs reinforcement learning with a pixel-similarity reward computed over unedited regions and argues that multimodal LLMs are not enough to judge consistency. Its masks, however, are themselves MLLM-annotated and noisy. EditReward and EditScore build learned reward models for editing, and both admit a bias toward rewarding a successful foreground edit even when the background is damaged. EdiVal-Agent shows that open-vocabulary detectors (Grounding DINO) are needed to check that unedited objects stay unchanged, but it only evaluates and does not train. All of these use direct instructions. None of them varies how indirectly the target is specified, which is the variable we control.
The multi-hop direction supplies both the mechanism and the recipe. IRCoT and ITER-RETGEN show that one-shot retrieval fails when the query lacks the intermediate "bridge" entity. We treat an editor's spatial attention the same way: when the referent is unresolved, attention cannot land on the right region. Learning from Synthetic Data Improves Multi-hop Reasoning shows that rule-generated, fully verifiable fictional multi-hop data (PhantomWiki) teaches a composition skill that transfers to real QA. We borrow that grammar-generated, verifiable-by-construction design and move it into scene space.
The data-filtering direction supplies the principle. Promptagator and CATfOOD filter synthetic text data with round-trip checks. GEMeX-RMCoT shows that 202K consensus-filtered samples beat 1.59M unfiltered ones. We replace consensus among fallible judges with a check against a program whose answer is known.
The strongest constraint on this proposal is Agent Banana. Its planner resolves the edit and then edits local crops, which gives strong background fidelity. We have to beat that kind of pipeline on distractor leakage and on physical coherence, not just match its preservation scores.
Research Program
Building the scenes and instructions. We build a scene-program generator over two image sources. The first is procedurally rendered scenes, where object identity, attributes, and spatial relations are exact. The second is real COCO-style images, the source MagicBrush uses, with object pools and relations extracted by Grounding DINO in the way EdiVal-Agent does. Only the rendered scenes are exact; for the real images, the program is exact only relative to the extracted object pool, which is itself detector output. A grammar samples referring chains of one to four hops over this graph, for example target = holds(left_of(person, window)). It pairs each chain with an attribute edit and also emits the direct instruction, such as "recolor the red mug." Scenes are sampled so that each target has at least one same-category distractor.
Generating candidate edits. Strong open editors, such as Qwen-Image-Edit, produce candidate edits from both the indirect and the direct instruction. Using the direct version gives many more clean candidates. The student is later trained to map the indirect instruction to the same output.
Filtering. A sample is kept only if three conditions hold. First, non-target-region PSNR and SSIM stay above a threshold outside a dilated target mask. The dilation leaves room for shadows and reflections. Second, every distractor passes a detector-and-crop invariance check. Third, the target's attribute change is verified. For multi-turn chains, the check runs on each hop, which addresses the single-step limitation of current filters.
Training. The student is fine-tuned on the kept data, with indirect instructions as input. As a secondary variant, we add CoCoEdit-style region rewards, now computed with program masks instead of MLLM masks.
Baselines
Naive reference points. The do-nothing baseline is the off-the-shelf editor given indirect instructions. The unfiltered baseline trains the student on every generated candidate. Both use the same image pool, the same number of training steps, and the same student.
Learned-reward filtering. The strongest data-filtering competitor is EditReward used as the filter. We keep the same fraction of data it kept in its own study, which is about 20K of 46K, and that raised GEdit-Bench from 6.7 to 7.1 for Step1X-Edit. We also run EditScore with Avg@4 self-ensembling as a filter. It reports 0.763 accuracy on EditReward-Bench. Beating these means lower distractor leakage at an equal or higher instruction-following score, with the kept-set size matched to ours.
Region-reward reinforcement learning. The strongest training-time competitor is CoCoEdit, run with its MLLM-annotated masks on our instructions and with the same reinforcement-learning budget. It reports +2.8 dB non-edited-region PSNR for Qwen-Image-Edit on GEdit-Bench.
Resolve-then-edit pipeline. A vision-language model rewrites the indirect instruction into a direct one, and an Agent Banana-style crop editor executes it. Agent Banana reports SSIM-OM 0.84 and LPIPS-OM 0.12 on HDD-Bench. Here we use an open planner, not GPT-5-mini, so that the compute is comparable.
Ablations. The key ablation swaps program masks for MLLM-annotated masks inside our filter while leaving everything else fixed; this isolates the value of exact masks. A second ablation trains on the same images using direct instructions only, which isolates the value of the multi-hop content.
For multi-hop referring edits, no reported result found on Lacuna, so the benchmark numbers on our test set will be new.
Why not just…?
Why not just crop and inpaint with a hard mask? A hard mask does guarantee that outside pixels stay untouched, as NEP and Agent Banana show. It has two weaknesses here. It requires the right mask, which is exactly what multi-hop instructions fail to supply. It also breaks shadows, reflections, and perspective, as noted in CoCoEdit and ChronoEdit. Our dilated-mask filter allows coherent lighting changes near the target. A human plausibility study on add-object edits will show whether this matters in practice.
Why not just have a VLM rewrite the instruction first, then edit? This is the real threat. If a zero-shot VLM resolves the reference reliably, then a trained single-pass editor adds little beyond speed. We test this directly by measuring resolver accuracy against program ground truth as hop count rises. If rewriting stays accurate at three and four hops and the pipeline matches our distractor-leakage numbers, we will report that. In that case the contribution narrows to the diagnostic and the program-verified filter, which can then be used to train the resolver instead.
Why not just filter with a strong VLM or reward model? EditReward and EditScore document their own background over-editing bias, and CoCoEdit shows MLLMs miss subtle drift. Our mask-swap ablation and the EditReward-filter baseline measure how much leakage those filters let through.
Why not the round-trip idea of inverting the edit chain back to the original image? Inverse edits are themselves lossy. A failed round trip cannot tell forward leakage apart from an imperfect inverse. Program masks avoid this confusion entirely.
Proposed Experiments
The first decisive check is diagnostic. On the held-out program test set (one to four hops), we measure non-target PSNR and distractor-change rate as a function of hop count for off-the-shelf editors. If leakage does not rise with hops, or does not concentrate on distractors, the central hypothesis fails.
The second check compares our student against all baselines on the same test set. It uses distractor-change rate, non-target PSNR, SSIM and LPIPS, and a detector-verified target success rate. On a human-rated subset, it also uses EdiVal-style consistency, which has 81.3% agreement with human judgment.
The third check guards against regressions and tests transfer. We report GEdit-Bench, EdiVal-Bench three-turn sessions, and MagicBrush, and we check whether training on rendered scenes alone transfers to real images, mirroring PhantomWiki's fiction-to-real transfer.
The final check is hop generalization. We train on one to two hops and test on three to four. If gains hold at unseen depths, the student has learned to compose references rather than memorize templates.
References
- CoCoEdit: Content-Consistent Image Editing via Region Regularized Reinforcement Learning
- EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing
- EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling
- EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
- Learning from Synthetic Data Improves Multi-hop Reasoning
- Promptagator: Few-shot Dense Retrieval From 8 Examples
- CATfOOD: Counterfactual Augmented Training for Improving Out-of-Domain Performance and Calibration
- GEMeX-RMCoT: An Enhanced Med-VQA Dataset for Region-Aware Multimodal Chain-of-Thought Reasoning
- Agent Banana: High-Fidelity Image Editing with Agentic Thinking and Tooling
- NEP: Autoregressive Image Editing via Next Editing Token Prediction
- ChronoEdit: Towards Temporal Reasoning for In-Context Image Editing and World Simulation