When Editing Agents Believe Their Own Reports: History Anchoring in Multi-Turn Visual Verification

Image editing agents now work in loops. A user gives a complex instruction such as "remove the cup, make the lamp brighter, then move the cat to the sofa." The agent splits it into small edits, calls an image editor for each one, looks at the result, and decides whether to continue or stop. Each "look" is done by a vision-language model acting as a judge. That judge usually sees the whole conversation, which includes the agent's own earlier statements such as "the cup has been removed." We propose to test a specific and consequential hypothesis. Multi-turn visual judges anchor on these self-reports instead of on the pixels. They confirm edits that never happened, and the error grows as the conversation gets longer. This is the multi-turn form of a known failure: models answer chart questions from the chart's title instead of from its data. Here the agent's own narrative plays the role of the title.

The core proposal has three parts. First, we build a controlled benchmark of multi-turn editing trajectories in which the textual history and the visual truth are deliberately set against each other. Second, we measure how often the judge's verdict flips when only the narrative changes. Third, we test a simple structural fix: a verifier that conditions on what the user asked for at every turn but never on what the agent claims to have done. We then add a trained version of that verifier and check whether better verification translates into fewer premature stops and fewer over-edits in a real editing agent.

Core Claim

Our claim is that, in closed-loop multi-turn editing, verification errors are driven substantially by conditioning on the agent's self-reported outcomes. We expect these errors to be largest where visual evidence is weakest, which means removals, subtle attribute changes, and later turns. We further claim that separating intentions (the user's instructions, which a verifier legitimately needs in order to resolve references like "make it bigger") from claims (the agent's statements about what happened, which it does not need) removes most of this error. It does so without the loss of reference resolution that comes from blindly deleting the history.

Three strands of prior work motivate the claim, and none tests it. MIRA builds exactly the kind of loop we study: a Qwen2.5-VL agent that inspects each intermediate image and emits a stop signal. The paper names over-editing and premature stopping as open problems but does not ask whether its inspector is biased. EditInspector shows that vision-language judges of single edits are "yes-men" and that hallucination rates for removal instructions are nearly double those for additions. Its authors explicitly leave multi-turn editing open. Mitigating Perceptual Judgment Bias documents a "plausibility trap," in which judges reward fluent text that contradicts the image, and trains a judge on counterfactual rankings. That work is single-turn and covers math and science diagrams only. We carry its counterfactual method into sequential trajectories, where the misleading text is the agent's own history rather than a candidate answer.

Two further papers shape the measurement design. The Silent Judge measures shortcut bias through a verdict shift rate, the fraction of verdicts that change when only a superficial cue changes. It finds a 0% rate at which judges acknowledge the cue in their explanations. We reuse both measures, with the cue replaced by the self-report. AgentEval shows that 63% of step failures in text and code agents propagate from upstream errors, and that evaluating each step against its immediate upstream context raises failure-detection recall from 0.41 to 0.89. That work contains no visual component. Our finding could complicate its recommendation: in the visual setting, upstream context may be exactly what corrupts the step-level verdict.

Research Program

The first component is a trajectory benchmark with ground truth fixed by construction. We start from multi-turn instruction sequences drawn from three sources: the four-round Multi-turn PIE-Bench extension of Multi-turn Consistent Image Editing, MagicBrush-style edits as used in EditInspector, and the compositional instructions of the MIRA benchmark. We run real editors on these sequences and record trajectories with the agent's self-reports.

We then create counterfactual variants that change only one factor, either the image or the narrative. Visual failures come in three forms. A null edit returns the pre-edit image. A masked partial edit reverts the target region. A deliberate collateral artifact is pasted outside the target region. Because we create these failures ourselves, their labels are known without human annotation. Narrative variants swap a truthful "the edit failed" for "done," or insert a confident description of the intended change. A human-checked subset validates that the real trajectories are labeled correctly.

The second component is a set of diagnostic measurements. The narrative verdict shift rate is the change in the judge's "edit succeeded" rate when the self-report flips from failure to success while the images stay the same. We also track the cue acknowledgment rate, meaning whether the rationale ever cites the history as its reason. We break both results down by turn index, by operation type (add, remove, attribute, or spatial), and by whether the judge is asked to reason step by step before answering. The last factor directly tests the "chain-of-thought paradox" reported in MLLM-as-a-Judge, where step-by-step reasoning led judges to hallucinate premises. That paper also found that judges sometimes do better with text descriptions than with images. Our prediction is the opposite of what that finding would suggest in this setting: added text should hurt here, because the text is self-serving.

The third component is the verifier itself. Its input is the original image, the previous image, the current image, a visual difference view in the style of EditInspector's difference triplets, and a ledger of user instructions up to turn $t$. The ledger resolves references, so "it" still points to the cat. The verifier never receives agent outputs. We evaluate this verifier first as a prompting change on off-the-shelf judges. We then train a 7B version with group-relative policy optimization, the reinforcement-learning recipe used by Perception-Judge, on counterfactual trajectories in which narrative and truth are statistically independent. The reward is verdict correctness, plus a ranking term that orders a correct edit above a partial edit and a partial edit above a null edit. This mirrors the quadruplet ranking in the perceptual-bias paper. Finally, we place the verifier inside the MIRA loop in place of its inspector and measure the downstream effect on editing.

Baselines

The naive reference point is MIRA's own inspector exactly as deployed: Qwen2.5-VL with the full conversation history. This is what current agents actually do. MIRA reports a +13.4% semantic-consistency gain with Flux.1-Kontext, averaging 4.1 iterations. We will reproduce that setup and report its narrative verdict shift rate and its downstream editing score.

The strongest existing judge for perceptual grounding is Perception-Judge from Mitigating Perceptual Judgment Bias. It reports 85.7% accuracy at identifying perceptual errors on the PPJD counterfactual set, against 69.5% for base Qwen2.5-VL-7B. We run it on our trajectories in two conditions, with full history and with the instruction ledger, so that any gain we show cannot be credited merely to a better single-turn perceiver. For edit inspection specifically, the strongest reported numbers come from EditInspector: 70.3% basic inspection accuracy for Gemini 1.5 Pro and 65.7% artifact-detection accuracy for GPT-4o. We run GPT-4o with EditInspector's difference-triplet prompting as the strong closed-model competitor. We find no reported multi-turn verification accuracy on Lacuna for any of these judges, so the multi-turn numbers will be established here.

The history-stripped judge sees only the previous and current images and the current instruction, with no ledger. We also evaluate a CLIP-direction and DINO-similarity threshold verifier of the kind used in Multi-turn Consistent Image Editing. These two isolate the key ingredient. If history stripping matches our ledger verifier even on turns with pronoun or reference dependencies, the ledger adds nothing. The trained verifier must also beat an untrained ledger prompt applied to the same 7B base model. All trained variants use the same base model, the same trajectory pool, and the same number of policy-optimization steps and tuning sweeps.

Beating the baselines means two things. First, the narrative verdict shift rate falls to within noise of zero while accuracy on reference-dependent turns is at least as good as the full-history judge. Second, inside MIRA, the rates of premature stopping and over-editing on the MIRA benchmark fall significantly relative to the stock inspector, at equal or lower iteration count.

Why not just…?

Why not just delete the history from the judge's context? This is the most serious threat, and it may be most of the answer. It is cheap and removes the cue entirely. Its cost is that later instructions often refer to earlier ones, so a history-blind judge cannot tell which object "it" is or whether "undo the lighting change" was satisfied. The history-stripped baseline, reported separately on reference-dependent and reference-free turns, settles this. If stripping ties the ledger on both, we will report that the fix is trivial and that the contribution is the measurement and the downstream effect on the agent.

Why not just use a stronger or already debiased judge such as GPT-4o or Perception-Judge? Stronger single-turn perception does not by itself stop a model from deferring to confident text. The Silent Judge finds GPT-4o shifting verdicts by up to +30% on a recency cue while never acknowledging the cue. Perception-Judge was trained only on single-turn candidate answers, never on self-generated histories. Running both under full-history and ledger conditions shows directly whether their robustness transfers.

Why not just use CLIP, DINO, or pixel metrics, which have no narrative to anchor on? They cannot be fooled by text. However, EditInspector and I2EBench document that these metrics miss localized edits, removals, and lighting consistency, which are exactly the cases where we expect anchoring to be worst. We include them as a verifier inside the MIRA loop. If they match our verifier on removal turns, that result would argue for hybrid verification rather than a learned judge.

Why not caption the images and let a strong text reasoner judge? Decoupling perception from reasoning works for math, as shown by RAPID. But a caption-based judge turns the whole problem into text, and text is where the self-report lives. Plot Twist shows accuracy falling from about 79% to the 30% range once textual shortcuts are removed. We run a caption-then-reason judge on our counterfactuals and predict it will show the highest verdict shift rate of any condition.

Decisive Checks

The first check determines whether the project continues. If off-the-shelf judges show a narrative verdict shift rate of only a few points on our counterfactual trajectories, the anchoring hypothesis is false. We would then report the benchmark as a negative result and not proceed to training. The second check is structural. If the effect is real, we predict that it grows with turn index and is largest for removals, consistent with EditInspector's removal gap. A flat profile would suggest generic judge noise rather than anchoring on history. The third check tests whether better verification matters at all. On the MIRA benchmark and on the verdict and rectification scores of R³-Bench, where R³-Refiner reports gains of +12.0% and +9.0% respectively, swapping in our verifier must reduce premature stops and over-edits without increasing iterations. If verification accuracy improves but editing outcomes do not, the bottleneck lies in the editor rather than the judge, and we will say so.