When a vision-language model (a system that answers questions about images) does well on an old benchmark, we cannot tell whether it is looking at the image or recalling a test item it saw during training. Existing contamination tests for these models mostly perturb the text: they shuffle answer options or paraphrase captions. They never ask the most direct question. If we delete the very object the answer depends on, does the model still give the old answer?
We propose to measure that stale answer rate: how often a model still gives the original-image answer after the answer-critical object has been cleanly erased. The test compares two kinds of images. One set comes from old, widely scraped benchmarks. The other is a matched set of fresh, post-cutoff photographs with the same object classes, scene types, and object sizes. Both sets carry the same everyday co-occurrence priors ("tennis courts have balls"), so a model relying on priors should behave the same on both. Any extra staleness on the old images is evidence that the model recognizes the specific image from its untouched surroundings and retrieves a memorized answer.
The test is only as good as the erasure. That is where the second research direction becomes essential. Standard image-quality metrics cannot certify that an object was removed. So the probe has to be built on reference-free checks for erasure validity, plus a sham-erasure control.
Core Claim
Counterfactual object erasure, applied to matched old and fresh images and checked by reference-free validity tests, yields a contamination signal that text-only probes cannot produce. The signal works at the instance level. It separates three things: memory tied to a specific image, memory carried by the question text alone, and generic language-prior bias.
We will validate this against known ground truth. We deliberately contaminate an open model on a random half of benchmark items, then ask whether the erasure signal identifies those items better than existing probes do.
The quantity we report for a model $f$ is
$$G = \big(S_{\text{old}}^{\text{erase}} - S_{\text{old}}^{\text{sham}}\big) - \big(S_{\text{fresh}}^{\text{erase}} - S_{\text{fresh}}^{\text{sham}}\big),$$
where the terms are defined as follows. $S^{\text{erase}}$ is the rate at which $f$ returns the original-image answer after the answer-critical object is erased. $S^{\text{sham}}$ is the same rate after an equally sized, object-free background region is erased with the same inpainter. The subscripts mark the image source.
The sham term cancels the effect of the inpainting itself. The old-minus-fresh contrast cancels prior-driven "yes-bias." What remains, $G$, is attributable to memory of specific old items.
Why this sits between the two directions
Erasure-based diagnostics already exist. ROHE erases COCO objects with the LaMa inpainter and shows that models keep claiming removed objects are present. MERLIM finds an average precision drop of about 8% under inpainting and calls the masked failures "hidden hallucinations."
Both papers motivate this proposal, but they draw a conclusion their designs cannot support. Both attribute residual "yes" answers to language priors. Yet ROHE is built entirely on MSCOCO 2017. MM-DETECT reports that older benchmarks such as COCO and ScienceQA leak more heavily than newer ones. So part of what these papers call prior-driven hallucination may actually be memorized image–answer pairs. Neither paper has a fresh-image control that could tell the two apart.
MM-DETECT and CAP supply the contamination-detection logic we reuse. Both compare performance before and after a perturbation. CAP has confirmed cases: it flagged Baichuan-13B on FinEval, later traced to a 21-question verbatim overlap. Their perturbations are purely textual, however. MM-DETECT itself finds that leakage often enters through the language backbone. A text perturbation therefore cannot tell us whether the image played any part in the recall.
MMBench-Live avoids contamination by fetching fresh web images. Its authors concede that fresh images still allow implicit memorization of common scene archetypes. We use fresh images differently: as a matched control, not as the benchmark itself. The matching is exactly what lets us subtract archetype effects.
The object-removal literature constrains how the probe can be built. The diminished-reality survey explains why no physical ground truth exists for a scene with the object removed. Classwise object-removal evaluation shows why full-reference metrics fail here. These metrics score a generated image by its pixel distance to a reference: peak signal-to-noise ratio, structural similarity, and the learned perceptual similarity LPIPS. In object removal they reward a model that removes nothing.
Erasers also invent replacement objects, which EntityErasure calls "sundries." It reports mean sundries per image of 0.554 for PowerPaint and 0.679 for CLIPAway, against 0.122 for its own method. EditInspector adds a further constraint: VLM judges hallucinate nearly twice as often on removal edits as on addition edits. A VLM therefore cannot be trusted to certify our erasures.
We take our validity checks from ReMOVE and EntityErasure. ReMOVE compares vision-transformer features inside and outside the mask and agreed with human preferences 74.7% of the time, versus 71.9% for LPIPS. EntityErasure contributes two sundry metrics: mean sundries count and mean sundry area ratio.
Research Program
Matched image pairs. We first build old/fresh pairs at the object level. Old items come from ROHE's COCO val2017 pairs and from the natural-photo subset of ScienceQA. ScienceQA matters here because MM-DETECT reports severe contamination for Claude-3.5-Sonnet on it.
Fresh items are photographs posted after the training cutoffs of the evaluated models, collected through the Flickr-style acquisition that MMBench-Live describes. Each fresh image is matched to an old one on four properties: target object class, scene category, the target's relative area, and the set of co-occurring object classes. We report how well the matching holds. A text-only language model's prior for "is there an X?" given a scene caption must be statistically indistinguishable between the two sets. This is our quantitative check that the priors are equal.
Certified erasure. Each target is erased by two inpainters: LaMa, to stay continuous with ROHE, and EntityErasure. An erasure is accepted only if it passes four checks:
- ReMOVE, computed on a local crop. Crops are needed because the ReMOVE paper's own ablation shows uncropped scores become erratic when the mask is small.
- Zero sundries, measured by an open-vocabulary detector run inside the mask.
- The original object class falls below detector threshold.
- A human audit of a stratified sample, used to estimate the false-accept rate of checks 1–3.
Pixel-distance metrics are reported only to show they rank the identity map highest. Every erased image gets a sham twin: a same-size background region at a matched location is erased with the same inpainter and passes the same checks.
Measuring staleness. Questions follow ROHE's presence/absence format and MERLIM's counting and relation formats. Each question is asked in five paraphrases. We take a majority vote across them, because MERLIM finds rephrasing alone can swing accuracy by up to 26.5%.
Is the memory keyed to the image? To test whether staleness depends on recognizing the specific image, we apply an identity-breaking but content-preserving transform to the erased image: a horizontal flip, a mild crop, or a color shift. If staleness on old images falls under this transform while staleness on fresh images does not, the model is recognizing exact images rather than scenes.
Baselines
The naive reference point is ROHE-style evaluation as currently practiced. The model is scored on erased COCO images with no fresh control and no validity filter, and every residual "yes" is read as prior-driven hallucination. On this protocol, ROHE reports an acc+ of 39.21% for LLaVA-1.5-7B, rising to 61.65% after object-based preference optimization. It also reports LLaVA-1.5-13B often below 30% on absent objects. We reproduce these numbers first, on the same 5,504 pairs, to anchor the comparison. Our proposal beats this reference point if it can show that some fraction of ROHE's residual errors is memorization (a positive $G$). It also beats it if a model's ranking under $G$ differs from its ranking under raw acc+.
The strongest existing detector is MM-DETECT's pair of text perturbations: option-order sensitivity for multiple-choice items and slot guessing on perturbed captions. CAP-style paraphrase consistency serves as a second text-only competitor. These are the strongest competitors because they are the only published multimodal contamination probes, and CAP has a verified detection. We found no reported number on Lacuna for MM-DETECT's instance-level detection accuracy against known ground truth, so we will measure it ourselves.
The fair comparison uses a controlled-contamination model. LLaVA-1.5-7B is fine-tuned on a random half of the old items, each given as image plus question plus answer, with the other half held out. Every detector receives the same items, the same query budget per item, and thresholds tuned on the same small validation split. Our method wins if it achieves higher area under the detection curve at separating contaminated from clean items. The crucial case is a second condition where contamination is injected with image–answer pairs whose question wording differs from the test wording. There, text probes should fall toward chance, while erasure staleness should not.
The blind baseline asks the question with no image or a blank image. It isolates text-only memory, and $G$ must add information beyond it on the same items.
The ablation that isolates our key ingredient removes the erasure-validity filter. We recompute $G$ with uncertified LaMa erasures and with sham correction switched off. This shows how much apparent memorization or hallucination is really inpainting residue.
Why not just…?
Why not just use text perturbations as MM-DETECT and CAP do? MM-DETECT's own finding that leakage enters through the language backbone means its probes detect text memory. They cannot say whether the image triggered recall. The second contamination condition settles this directly: it contaminates with re-worded questions so that text cues are absent. If text probes still match our signal there, our case collapses, and we will report that outcome.
Why not just ask the question with a blank image? This is a serious threat for items whose answers are recoverable from the question text. A blank image, however, also deletes the context that would let a model recognize a memorized photo. A blank image removes the whole picture; erasure removes only the evidence, so image-keyed memory survives. The difference between staleness under erasure and accuracy under a blank image is the quantity at stake. If that difference is near zero on the controlled model, the image-recognition pathway does not exist and erasure is unnecessary.
Why not just paste a black box over the object? This is cheap and has no artifacts, and it also preserves the surrounding context needed for image recognition. It could plausibly work, so we run it as a full arm rather than dismiss it. Our concern is that a black box signals occlusion and pushes models toward "can't tell" responses on both old and fresh images. That would compress the stale-answer range and lower detection power. If the black-box arm matches certified inpainting on detection, we will adopt it and treat the erasure-validity machinery only as a correction to prior hallucination benchmarks.
Why not just evaluate on fresh images and skip detection? MMBench-Live already does this. It produces a cleaner benchmark but says nothing about which existing results were inflated, and its authors note that archetype memory remains. Our matched design uses fresh images to subtract archetype memory, not to escape it.
Proposed Experiments
Experiment 1: controlled contamination. On the controlled-contamination LLaVA-1.5-7B, we report detection performance for $G$-based instance scores, MM-DETECT, CAP-style paraphrase, blind, and black-box probes. We use both verbatim and re-worded contamination conditions. A decisive positive result has three parts:
- Erasure staleness clearly beats text probes in the re-worded condition.
- The identity-breaking transform shrinks staleness only on contaminated items.
- Sham-corrected $G$ on the uncontaminated half is near zero.
Experiment 2: real models. We compute $G$ for public open models (LLaVA-1.5 7B/13B, Qwen2.5-VL-7B) and, where API access allows, for proprietary models on COCO and the ScienceQA photo subset. We then re-score ROHE and MERLIM-style results after removing items with high instance-level staleness. The question is whether the "hidden hallucination" rates those papers report shrink, and by how much.
Experiment 3: erasure validity. We report the human-audited false-accept rate of each validity check. We show how LaMa's uncertified erasures shift measured hallucination relative to certified ones, testing whether residual silhouettes inflate "yes" answers. The reference point is EntityErasure's reported sundry rates, in which PowerPaint and CLIPAway leave measurably more sundries than EntityErasure.
Failure modes we will report. Either of two outcomes would falsify the core claim. One is $G$ indistinguishable from zero on the deliberately contaminated model. The other is the blind or black-box probes matching erasure on every condition.
References
- Evaluating and Mitigating Object Hallucination in Large Vision-Language Models: Can They Still See Removed Objects?
- Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models
- Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLMs
- CAP: Data Contamination Detection via Consistency Amplification
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models
- A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects
- Object Remover Performance Evaluation Methods Using Classwise Object Removal Images
- EntityErasure: Erasing Entity Cleanly via Amodal Entity Segmentation and Completion
- EditInspector: A Benchmark for Evaluation of Text-Guided Image Edits
- ReMOVE: A Reference-free Metric for Object Erasure
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering