Vision-language models can be corrected at inference time by pasting the new fact into the prompt. For example, a model can be told "this jersey's crest belongs to Inter Miami" and asked about an image, with no weight updates. The strongest recent editors of this kind report near-perfect success on the edited questions. Every one of those success numbers, however, asks only whether the model says the new answer. None asks whether it said it because of what is in the image. A separate literature shows that multimodal models routinely answer from the words alone, and that public image benchmarks are partly memorized. Together these raise a concrete worry: an edit that "works" may just be a sentence the model copies whenever the question text looks similar, regardless of the picture.
We propose to measure this directly and then to fix it. The measurement uses visual twins. Each edit is paired with an image where it should apply and a near-identical image where it should not, and edits are also tested with the image removed. Twins are built from image regions the base model has provably not memorized. The fix is a small change to how edits are injected. The model first answers, from the image alone, whether the edit's visual precondition holds, and only then receives the edit. The central claim is that current multimodal in-context editors have high reliability but low image selectivity. They apply edits to the wrong twin nearly as often as to the right one. Evidence-first injection closes much of that gap without losing reliability.
Core Claim and What Is New
In-context knowledge editing means supplying updated facts in the prompt so the model overrides what it learned in pre-training. It was formalized for text by Zheng et al. and extended to images by three systems. ReasonEdit retrieves stored human rationales. ComprehendEdit / HICE puts a classifier gate in front of retrieval that decides whether a query is in scope. CARML weights visual, textual and fused retrieval channels and corrects output logits.
All three evaluate reliability (the edited question gets the new answer), generality (paraphrases do too), and locality (unrelated queries are untouched). Locality is tested with unrelated queries, not with the same question over a visually different image. That is precisely the case where a text shortcut and genuine grounding make different predictions.
Our contribution is an edit metric that separates the two, together with an injection method designed to score well on it. Image-conditional selectivity is defined as
$$S = P(\text{edited answer} \mid I^{+}) - P(\text{edited answer} \mid I^{-}),$$
where $I^{+}$ is an image satisfying the edit's visual precondition and $I^{-}$ is its twin that does not. A pure text shortcut gives $S \approx 0$. Correct grounding gives $S$ near reliability. We also report the blind-edit rate, which is edit success with the image replaced by a blank. This follows the unimodal ablation logic of Thomason et al. and MMStar.
Why These Literatures Motivate the Worry
Three lines of evidence make low selectivity likely.
The first is that text alone often suffices. MMStar found GeminiPro scoring 42.9% on MMMU without images. MathVerse found some models do more than 5% better when the diagram is removed, because descriptive text in the prompt substitutes for looking. An injected edit or rationale is exactly this kind of descriptive text.
The second is that models can score well while attending to irrelevant regions. CofactGVR showed that a baseline grounded-reasoning model stayed flat or improved when forced onto random crops, while its causally trained model dropped 78.7 points on MME. That result supplies our intervention logic: compare the correct image against a counterfactual one. Unlike CofactGVR, we require no reinforcement-learning fine-tuning and target edits rather than general grounding.
The third is that memorization confounds everything above. Contamination Detection for VLMs Using Multi-Modal Semantic Perturbations contaminated LLaVA-v1.5-7B for 3 epochs. The model rose from about 38% to about 54% on original MMStar but fell from about 69% to about 38% on perturbed images. Editors are evaluated on public sets such as A-OKVQA and FVQA, where memorized image-answer associations could either fight the edit or make it look easy. We reuse that paper's perturbation pipeline to build twins that are off the memorized distribution.
DecKER is a constraining precedent and a methodological one. In text-only models it showed that injecting conflicting facts mid-reasoning can collapse multi-hop accuracy by over 80%. It fixed this by building a reasoning skeleton before inserting facts. Our injection method borrows the ordering idea: settle the perception first, then add the edit. The difference is that we decouple perception from edit rather than reasoning structure from entities.
Research Program
The first component is a twin-based edit test set. Edit records come from ComprehendEdit, the MMEdit E-VQA subset, and VLKEB. For each edited image $I^{+}$, a twin $I^{-}$ is made by changing only the visual attribute the edit depends on, such as the crest, the object count, or the spatial relation. The change uses the pipeline of the contamination paper ( Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations): dense captioning, Flux with ControlNet edge maps, and a solvability filter.
Diffusion artifacts are a known risk. To guard against them, we add human-captured counterfactual pairs from NaturalBench, where the twin exists naturally. We also report every metric separately for synthetic and natural twins. A twin is valid only if the unedited base model answers the attribute question correctly on both images, which confirms the difference is visible to the model.
The second component is contamination stratification. For each source item, we compute the base model's gap between its original-image and perturbed-image accuracy, as in the contamination paper. Items are split into likely-memorized and likely-clean strata. Every result is reported per stratum, so we can say whether apparent editing success is concentrated where the base model memorized the item.
The third component is evidence-first injection. Given a query and a retrieved edit, the model is first asked a short perceptual question derived from the edit's precondition, such as "what crest is on the jersey?", with the edit withheld. The edit is then injected together with the model's own perceptual answer. The edit is framed as conditional ("if the crest is X, then …"). Retrieval and scope gating are otherwise identical to the host editor, so the method can be layered onto IKE, HICE, or CARML. No weights change.
Baselines
The naive reference is the unedited base model, together with plain IKE prompting of the retrieved edit (Zheng et al.). Plain IKE is the pure in-context condition against which all selectivity numbers are read. Its known weakness on standard locality is documented in ComprehendEdit, where HICE holds about 81–83% multimodal locality while IKE often drops below 40% (ComprehendEdit).
The strongest existing competitors are HICE and CARML. Their scope gates are the one component that could already produce selectivity, if the gate reads image features. CARML's reported strength is reliability. On LLaVA-V1.5 it holds 99.3% after 1,000 sequential edits, where MEND falls below 1%. It reaches 94.5% on a mixed VQA and captioning stream versus 89.0% for LiveEdit (CARML).
ReasonEdit reports near-100% targeted-edit accuracy (ReasonEdit). We include it where its rationale codebook can be reconstructed for our edit sources. No reported result on Lacuna was found for selectivity or blind-edit rate for any of these editors; establishing those numbers is part of the contribution.
MEND is included only as a parameter-editing reference, run at a small number of edits so it is not trivially collapsed.
Fairness is enforced by running every method on the same backbones (LLaVA-1.5, BLIP-2, InstructBLIP, and Qwen3-VL where available), the same edit streams, and the same retrieval index. Each method gets the same hyperparameter search budget on a held-out split of non-twin edits. No method sees twins during tuning.
Two ablations isolate the key ingredient. The first swaps the perceptual pre-question for a text-only restatement of the precondition, which keeps the extra prompt length but removes the image-derived answer. The second keeps the pre-question but injects the edit unconditionally.
Beating the baselines means the following. Evidence-first injection must raise selectivity $S$ by a margin exceeding the seed-to-seed spread, on both natural and synthetic twins and in the clean stratum. It must also keep reliability within a few points of the host editor's own reliability.
Why not just…?
Why not just run the blind ablation, removing the image, on existing editors? This is cheap, and we run it. It only catches the extreme case where the model ignores the image entirely. An editor that notices "there is a jersey" and then applies the edit to any jersey passes the blind test but fails the twin test. Only $S$ distinguishes coarse image use from use of the attribute the edit depends on. If the blind-edit rate alone predicted $S$ almost perfectly across editors, we would report that and drop the twin machinery as unnecessary.
Why not just rely on HICE's or CARML's scope classifier? This is the most serious threat. Both gates exist to decide whether an edit applies, and CARML explicitly measures inter-modal conflict (CARML). If those gates already achieve high $S$, our method adds little. The experiment settles it directly: we measure $S$ for each gate as published. We also run a variant where the gate's input image is replaced by the twin while the query stays fixed. If gate decisions barely change, the gate is keyed on text. If HICE or CARML turn out to be selective, the proposal reduces to a measurement paper, and we would say so.
Why not just render the edit inside the image, so there is no text to shortcut? The vision-only setting of MMMU-Pro removes text shortcuts, but it pushed GPT-4o and Claude 3.5 Sonnet below 50%. It did so by overloading reasoning, even though the models could read the embedded text. We include in-image edits as a baseline and expect lower reliability. If they instead match reliability and achieve high $S$, that is a simpler fix, and it would supersede ours.
Why not just prompt chain-of-thought? Step-by-step prompting lowered accuracy for several models on MATH-Vision. On MathVerse, correct answers often came with hallucinated intermediate steps. Free-form reasoning gives the model more room to restate the edit, not a reason to check the image. We run a chain-of-thought variant of IKE with prompt length matched to evidence-first injection. This separates the effect of structure from the effect of extra tokens.
Proposed Experiments and Decisive Checks
The first experiment is the diagnosis. We report reliability, standard locality, blind-edit rate, and $S$ for every editor, backbone, and contamination stratum. The prediction that makes this work worth doing is that published editors show reliability near their reported levels but $S$ well below reliability. We further predict the gap is larger in the likely-memorized stratum.
A decisive negative outcome is possible: if existing editors show $S$ close to reliability on natural NaturalBench twins, the concern is unfounded for current systems. That would be a useful result in its own right.
The second experiment tests the fix. Evidence-first injection is applied on top of IKE, HICE, and CARML, alongside the two ablations. The decisive check is that $S$ improves only when the pre-question is answered from the image. The text-restated precondition ablation should not recover the gain.
The third experiment is an artifact control. On unedited clean models, accuracy on the twin attribute question should be comparable between synthetic and natural twins. The contamination paper's positive clean-model delta ( Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations) suggests synthetic renders are, if anything, easier. If synthetic twins behave differently from natural ones, we base our conclusions on the natural subset.
The fourth experiment probes the entanglement concern raised by DecKER. On a small set of two-hop visual edits drawn from VLKEB's knowledge-graph links, we check whether conditioning the edit on the model's perceptual answer reduces reasoning collapse. This probe is exploratory and is labeled as such.
References
- Can We Edit Factual Knowledge by In-Context Learning?
- ReasonEdit: Editing Vision--Language Models using Human Reasoning
- ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing
- Reliable Lifelong Multimodal Editing: Conflict-Aware Retrieval Meets Multi-Level Guidance
- Shifting the Baseline: Single Modality Performance on Visual Navigation & QA
- Are We on the Right Way for Evaluating Large Vision-Language Models?
- MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
- CofactGVR: Counterfactual Intervention for Grounded Visual Reasoning
- Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations
- Decoupling Reasoning and Knowledge Injection for In-Context Knowledge Editing
- Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations
- MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset