When a vision-language model scores well on a public benchmark, we cannot tell whether it looked at the image or recalled the item. Text-side contamination tests shuffle answer options or paraphrase questions. These tests are cheap, and a paraphrasing adversary can defeat them. The proposal is to probe the image instead. For benchmark items whose answer depends on one well-defined property of the image, such as the color of an object or which side of the frame something is on, we apply a deterministic edit that changes that property, and with it the correct answer, while leaving everything else fixed. A model that is really looking should change its answer. A model that has memorized the benchmark item keeps giving the old one, a "stale answer." Some models fail to see small edits for ordinary perceptual reasons, so we compare stale-answer rates on benchmark items against rates on freshly photographed items given the same edits. The excess on benchmark items is the contamination signal.
The feature-fusion literature contributes the measurement tool. Classical descriptors from different mathematical families respond to different physical properties of an image: color moments and opponent-color channels, edge histograms and Laplacian-of-Gaussian responses for shape, and wavelet and co-occurrence statistics for texture. A fused set of such descriptors can certify, without asking another neural model, which properties an edit changed and which it left alone. That makes the probe auditable in a way that edits made by generative models and scored by model judges are not.
Core Claim
Item-level recall of multimodal benchmarks leaves a behavioral trace that text perturbations cannot reach: the model is insensitive to answer-changing image edits on seen items, and more insensitive than on unseen items matched for edit type and edit size. We claim three things. First, this excess stale-answer rate detects controlled contamination at low false-positive rates. Second, it still works when the contaminating data was paraphrased, the regime where text-side detectors collapse. Third, it separates two sources of inflated scores that current tools conflate: text-only priors, where the model answers correctly without the image, and image-item recall, where the model uses the image but only to retrieve a stored answer.
We define the score as
$$S = \big(K_{\text{bench}} - K_{\text{fresh}}\big)$$
Here $K$ is the fraction of answer-flipped items on which the model repeats the pre-edit answer. The subscripts denote benchmark items and fresh control items. Both are restricted to items where the model answered the unedited version correctly and also passed an answer-preserving edit of the same kind. The restriction is what separates recall from blindness: a model that passes the preserving edit is demonstrably reading the image.
Positioning Against Prior Work
The closest work is Both Text and Images Leaked! (MM-DETECT). It detects multimodal contamination with option-order sensitivity and masked-caption slot guessing, and it traces leakage back to the text backbone. It treats the image as fixed. Its perturbations live in the text, so it cannot tell whether a model is exploiting a memorized image–answer pairing.
Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping (VLB) does edit images, adding, removing, or outpainting objects with PowerPaint. It relies on an InternVL-2 judge, and its edits are designed mostly to preserve the answer and make items harder. It therefore measures accuracy drops, not a signature of recall, and it inherits the biases of both the generator and the judge. VLB motivates this proposal in two ways: it reports that static benchmarks overlap pre-training image corpora by up to 84.46%, and that "Hard" variants lower MMBench accuracy by 5–10%.
MMBench-Live and KBE-DME replace or evolve benchmarks instead of diagnosing existing ones. MMBench-Live explicitly concedes that it cannot rule out implicit memorization of common visual concepts. We reuse its idea of fresh instances, but only as a control group.
On the text side, three papers set the method and the threat model. CAP supplies the method: it compares consistency on train versus test perturbations, and we borrow its comparative logic. Detecting Benchmark Contamination Through Watermarking supplies the evaluation design: it deliberately contaminates a model and measures detection statistically, a protocol we copy for images. Evading Data Contamination Detection for Language Models is (too) Easy constrains the design. It shows that paraphrase-based evasive training inflates scores by up to 15% while driving sample-level detectors to about 1% true-positive rate at 1% false-positive rate. Any credible new detector must be tested against that adversary.
From the fusion side, three papers supply the measurement tool. Color Descriptors for Object Category Recognition shows that intensity-based and color-invariant descriptors capture separable information. Multi Texture Analysis of Colorectal Cancer Continuum Using Multispectral Imagery shows the same for co-occurrence, Laplacian-of-Gaussian, and wavelet families. Content- based Image Retrieval using Fuzzy Integral shows the same for color, frequency, and edge descriptors on a spatial grid. We do not use these descriptors to classify anything. We use them to certify that an edit moved one channel and left the others still.
Feature Contamination plays a cautionary role. It shows that ReLU networks couple label-irrelevant background features into their representations. As a consequence, answer-preserving nuisance edits can change answers for reasons unrelated to memorization, which is exactly why the preserving-edit filter and the fresh controls are required.
How the Probe Is Built
We restrict attention to question types where a deterministic image transform has a known effect on the answer. This is a deliberate narrowing: the probe covers a subset of each benchmark, not all of it.
The first family is chromatic. For color-attribute questions, we segment the queried object and rotate its hue in opponent-color space to a target color that is one of the answer options. We then verify with color-moment descriptors that the object's color moved to the target, and with edge-histogram and Laplacian-of-Gaussian descriptors that shape did not change.
The second family is geometric. For left/right and relative-position questions, we mirror the image horizontally and apply the known remapping of answers. We exclude images containing text, maps, or asymmetric conventions, and a text detector filters them out.
Each flipping edit has an answer-preserving twin of matched magnitude, measured by the same fused descriptors. For a color question, the twin is a hue rotation applied to background regions only. For a spatial question, the twin is a wavelet-band texture resynthesis or a global hue shift. The twin verifies that the model is attending to the image and is robust to that kind of change.
Fresh controls are photographs taken for this project after the release date of every evaluated model, so none of them can be in any crawl. They are paired with questions written from the benchmark's own templates. Items are then matched to benchmark items by question type and by the descriptor-space size of the edit.
Source benchmarks are the attribute and spatial categories of MMBench, SEEDBench, and MMStar, plus the ScienceQA image questions that fit these templates (Learn to Explain describes ScienceQA's composition).
Research Program
The program has three stages.
The first stage is controlled contamination with ground truth. We start from an open model such as Qwen2.5-VL-7B-Instruct, the top performer reported by MMBench-Live. We fine-tune it on a random half of the probe-eligible benchmark items under four regimes: verbatim items; text-paraphrased items in the style of the evasive-augmentation adversary, with original images; images with captions but without questions; and no contamination. The held-out half supplies known-clean items from the same distribution. Exposure counts vary from 1 to 16, following the watermarking paper's dose design.
The second stage checks detection and decomposition. On each model, we compute item-level and dataset-level $S$, blind accuracy (the image replaced by a gray frame), and the text-side detectors. We then test whether the three signals separate the regimes. The expected pattern is as follows. Verbatim and paraphrase contamination should raise $S$. Caption-only contamination should raise $S$ weakly, if at all. Text-only priors should show up as high blind accuracy with low $S$.
The third stage is an audit of released models, including Claude-3.5-Sonnet on ScienceQA, which MM-DETECT flagged. We ask whether image-side and text-side evidence agree.
Baselines
The naive reference point is the benchmark's own accuracy with no probe. Alongside it we report accuracy under the edits without the fresh-control subtraction, which is roughly what VLB-style robustness numbers report. Beating this reference means $S$ ranks contaminated above clean items where raw edit sensitivity does not.
The blind baseline asks the model the question with no image. It is the cheapest serious competitor for detecting text-backbone leakage. It is run on identical items and models. We expect it to win on text-prior leakage and to be blind to image-item recall, since a model that recalls an image still needs the image. We count as success the case where $S$ detects paraphrase contamination with original images, which blind accuracy cannot flag. We also report if blind accuracy captures most of the signal, because that would demote the proposal.
MM-DETECT's option-order test and CAP's consistency ratio are the strongest existing detectors. They run on the same fine-tuned checkpoints, with the same items and the same number of queries per item, and we tune their thresholds on the same clean split we use for ours. The file contains no reported true-positive-rate figure for MM-DETECT or CAP on multimodal benchmarks: no reported result found on Lacuna for MM-DETECT detection rates. The relevant published reference point is the evasion paper's report of about 1% true-positive rate at 1% false-positive rate for sample-level detectors under paraphrase contamination on text benchmarks. Beating this baseline means a true-positive rate at 1% false-positive rate substantially above what option-order and CAP achieve on the paraphrase regime, with matched performance on the verbatim regime.
The VLB-style generative baseline edits the same items with inpainting and scores answers with a model judge. It uses the same query budget and asks whether generative, judge-scored edits yield a comparable excess over fresh controls. If it does, the descriptor-certified edits add only auditability, not detection power.
Three ablations isolate the key ingredients. The first removes the fresh-control subtraction. The second removes the preserving-edit filter. The third replaces descriptor verification with a judge model's verdict that the edit worked.
Why not just…?
Why not just remove the image and check whether the model still answers correctly? The blind test is strong, and we run it. It detects only leakage in which the image is unnecessary. The evasion paper's adversary, and the 84.46% image overlap VLB reports, both point to a regime where images are seen during training alongside answers. There, a model needs the image to trigger recall and will fail the blind test honestly. The caption-only and paraphrase regimes of the controlled experiment test directly whether blind accuracy misses this case. If it does not, the proposal shrinks to a confirmation of the blind test.
Why not just search for near-duplicate images in pre-training corpora? VLB does this for LAION-100M. The approach requires access to the corpus, which is unavailable for closed models. It also shows only exposure, not exploitation. Most of those overlapping images may never have been memorized together with their benchmark answers. $S$ measures exploitation behaviorally. The audit stage reports both measures where the corpus is public, to show how they differ.
Why not just use option shuffling or paraphrase consistency, as in MM-DETECT and CAP? These probe the text, and a paraphrasing adversary has been shown to defeat text-side sample-level detectors. Our paraphrase-contamination regime reproduces that attack on a multimodal model. This comparison is the decisive one. If the text detectors survive it in the multimodal setting, the image-side probe is not needed.
Why not just evaluate on fresh benchmarks like MMBench-Live instead of diagnosing old ones? Fresh benchmarks avoid the problem going forward but do not tell us which published numbers were inflated. MMBench-Live also admits implicit memorization of recurring concepts. We use fresh items only as the control group. The two approaches are complementary, and ours is the one that answers questions about existing leaderboards.
Proposed Experiments and Decisive Checks
The primary metric is item-level true-positive rate at 1% false-positive rate, reported separately for each contamination regime and exposure count. We also report area under the ROC curve and dataset-level $S$ with bootstrap confidence intervals.
The first decisive check is the paraphrase regime. Our claim holds if $S$ detects paraphrase-contaminated items well above the option-order and CAP detectors and above blind accuracy. Our claim fails if $S$ does no better there.
The second check asks whether $S$ rises with exposure count in a dose-dependent way. Recall should strengthen with repetition, so a monotone rise supports reading $S$ as recall. A flat curve would mean $S$ is tracking perceptual difficulty that the matching failed to control.
The third check is the ablation without the preserving-edit filter. It quantifies how much apparent staleness is really the background coupling that Feature Contamination predicts. If the filter removes most of the signal on clean models while contaminated models retain it, the design is working as intended.
Finally, the audit of released models reports where image-side and text-side evidence disagree. For example, a model might show high $S$ on ScienceQA color questions but a clean option-order score. Those disagreements are the cases current tools cannot see.
References
- Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLMs
- Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models
- Dynamic Multimodal Evaluation via Knowledge-Enhanced Benchmark Evolution
- CAP: Data Contamination Detection via Consistency Amplification
- Detecting Benchmark Contamination Through Watermarking
- Evading Data Contamination Detection for Language Models is (too) Easy
- Color Descriptors for Object Category Recognition
- Multi Texture Analysis of Colorectal Cancer Continuum Using Multispectral Imagery
- Content- based Image Retrieval using Fuzzy Integral
- Feature Contamination: Neural Networks Learn Uncorrelated Features and Fail to Generalize
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering