Vision-language models (systems that answer questions about images) are scored on public benchmarks whose images and answers are routinely scraped into their training data. If a model has memorized "this picture → answer B," a high score says nothing about whether it can see. The most direct visual test so far edits each benchmark image with a diffusion model (a generator that turns noise into images step by step) so that the correct answer changes, and checks whether the model follows the edit. That test has a problem. Clean models score about 31% higher on the edited images than on the originals. The edit changes how hard the task is, so it does not only isolate memory.
We propose a different probe. For each benchmark item, we retrieve real photographs that are not the benchmark image but would get the same answer to the same question: other photos of the same landmark, animal species, or object category. We call these siblings. We use visual consistency filtering to make sure each sibling truly belongs with the original. This is the classic web-data-cleaning idea that genuine instances of a concept form a tight cluster in feature space, while off-topic images sit apart as outliers. A model that understands the image should answer a sibling about as well as the original. A model that memorized the original's pixels will not. We then compare each model's drop from original to siblings against the drop seen in models known to be clean. This turns a noisy behavioral difference into a calibrated per-item contamination score that needs no generated images, no token probabilities, and no access to training data.
Core Claim
Retrieved real siblings, kept within a band of visual similarity, give a contamination signal that is better calibrated than diffusion counterfactuals on the questions where both apply. The band has two edges. A sibling must be close enough to sit in the concept's dense cluster, so the answer carries over. It must also be far enough that it is not a re-encoded copy of the benchmark image, which could itself be memorized. The new ingredient is this two-sided filter combined with a comparison against clean reference models. Without either piece, a retrieval-based swap reduces to an uncontrolled distribution shift. For a model $m$ and item $i$, let $a_m(x)$ be correctness on image $x$, $x_i$ the original image, and $S_i$ its sibling set. The score is
$$c_m(i) = \Big[a_m(x_i) - \tfrac{1}{|S_i|}\sum_{s\in S_i} a_m(s)\Big] - \operatorname{median}_{r\in R}\Big[a_r(x_i) - \tfrac{1}{|S_i|}\sum_{s\in S_i} a_r(s)\Big],$$
where $R$ is a pool of reference models believed clean on that benchmark. The second term absorbs how much harder or easier the siblings are than the original, which is exactly what the diffusion test leaves uncontrolled. A positive $c_m(i)$ that holds across items, checked with a paired test in the style of PaCoST, flags memorization.
Where This Sits
Three lines of work motivate the proposal by showing the gap.
Both Text and Images Leaked! audits twelve multimodal models with MM-DETECT, which shuffles answer options and masks words in back-translated captions. It finds that leakage often enters through the language backbone. It never changes the image, so it cannot see memory triggered by pixels.
Contamination Detection for VLMs using Multi-Modal Semantic Perturbation does change the image. It uses GPT-4o-written edit instructions and a Flux diffusion model guided by ControlNet. On MMStar, RealWorldQA and NaturalBench it reports that deliberately contaminated models drop 8–32% while clean ones gain about 31%. That gain is the uncontrolled difficulty shift we target. The authors also note that diffusion fails on small text and complex geometry.
Are We on the Right Way for Evaluating Large Vision-Language Models? builds MMStar by discarding questions that text-only models can answer without the image. That filter works only on text and is blind to memorized images.
The method comes from the visual-consistency literature, which we reuse rather than reinvent. Google Landmarks Dataset v2 cleans web images with a two-stage coherence check: global image features (ResNet-101) plus local keypoint descriptors (DELF). We use the same pair in two directions: global similarity to admit siblings into the concept cluster, and local keypoint matching to reject near-copies of the benchmark image. On-the-fly learning for visual search shows that pairwise consistency pruning over top web results runs in milliseconds. LEVAN shows that whether a quick classifier can learn a concept is a usable test of whether the concept is visual at all. We use that test to decide which benchmark items are eligible, meaning their answer is a visually repeatable concept.
Two papers constrain the design. Teterwak et al. show that CLIP image–label alignment is very high on images that carry readable text shortcuts. Siblings whose answer is literally written in the image would therefore be trivially easy, so we exclude them using an optical character recognition (OCR) check. MMBench-Live already retrieves fresh web images matched to benchmark patterns, judged by an agent built on a large model. Our contribution is not fresh images for a new benchmark. It is a paired, calibrated test on the existing benchmark, with images vetted by cheap, model-independent feature checks.
Research Program
The first component is eligibility. An item qualifies if its question remains answerable, with the same answer, under a change of instance. Examples are "what landmark is shown," "what kind of animal is this," and "what object is the person holding." For each candidate answer concept, we gather web images, preferring upload dates after the audited models' training cutoffs. Following LEVAN, we train a linear probe on frozen features. Concepts whose probe cannot separate them from background images are marked non-visual, and their items are excluded from the audit and reported separately.
The second component is the sibling filter. A candidate sibling is kept if four conditions hold. Its global embedding lies within the dense core of the answer concept's cluster, using a k-nearest-neighbor density threshold. It lies in that core for no competing answer option. It falls below a near-duplicate threshold in global similarity and shares few verified local-descriptor matches with the original. Its OCR output does not contain the answer string. We keep up to eight siblings per item and have humans audit a random sample for answer validity.
The third component is a second arm that retrieves answer-swapping siblings. These are real images from the cluster of a distractor option's concept, matched in layout to the original. This is a retrieval analogue of Park et al.'s answer swap without the generated pixels.
The fourth component is calibration and testing. The reference pool $R$ consists of the same base models fine-tuned on a split of instruction data that excludes the benchmark. For natural audits of released models, $R$ is a set of models with low leakage under MMStar's multimodal-leakage metric. Aggregating $c_m(i)$ with a paired t-test across items gives a dataset-level verdict. Thresholding $c_m(i)$ gives an instance-level flag.
Baselines
The naive reference point is raw accuracy with no audit, plus the image-withheld "blind" check from MMStar. These show how often contamination is invisible to accuracy and to text-only purification.
The strongest existing method is Park et al.'s diffusion-counterfactual test, which we rerun with their pipeline (GPT-4o instructions, Flux with ControlNet). We use the same items and the same contaminated and clean models, with an equal number of perturbed variants per item. Its reported result is that contaminated models drop 8–32% and clean models gain about 31% on perturbed MMStar, RealWorldQA and NaturalBench. No reported detection AUROC or false-positive rate was found on Lacuna for this method, so we compute both ourselves under identical conditions. Beating it means two things on the eligible subset. First, higher instance-level AUROC for separating contaminated from clean (model, item) pairs. Second, a lower false-positive rate on clean models, since the 31% shift should make a fixed threshold misfire across models of different strength. A secondary win is lower cost per item, with no generative model calls.
MM-DETECT (option shuffling plus caption slot guessing) is the strongest text-side detector. It runs on the same models with its published metrics: the change in correctness after perturbation and the per-instance flip rate. We expect it to miss contamination that is injected only as image–answer pairs.
Two ablations isolate the key ingredient. The first, "raw retrieval," uses the top-k web images for the answer concept with no density filter and no near-duplicate exclusion. The second keeps the filter but drops the reference-model calibration term. If raw retrieval matches the full method, the visual consistency filtering contributes nothing and the claim fails.
Why not just…?
Why not just use the diffusion counterfactuals that already exist? They are a real competitor and remain the only option for items whose answer depends on fine details that no real photo reproduces. We do not claim to replace them there. The evidence against using them everywhere is in their own paper. The roughly 31% gain for clean models means the edit changes difficulty, so a contaminated model's drop is measured against a moving baseline. The authors also report failures on small text and geometry. The head-to-head AUROC and false-positive comparison on shared eligible items settles which is better calibrated. If diffusion wins, the retrieval arm is useful only as a cheaper approximation, and we will say so.
Why not just apply standard augmentations such as crops, color jitter, or resizing? This is the cheapest visual perturbation and must be tested. We expect it to fail because memorization learned during training already sees augmented copies; Song et al. cite such augmentations as the reason exact overlap is hard to quantify. A model that memorized an image should therefore still recognize its augmented versions. Augmentations also probe only memory of exact pixels, not memory of an image–answer pairing at the level of the scene. We include augmentation as a baseline with the same number of variants. If it matches sibling retrieval in detection AUROC, the simpler method wins and we will report that.
Why not just score items with CLIP image–text alignment, as Teterwak et al. do? That score measures what a contrastive image-text encoder (CLIP) absorbed in pretraining, not what the audited model memorized. It is one number per item and is identical for clean and contaminated versions of the same model, so it cannot separate them in our controlled setup. We use it only as a filter for text shortcuts and report its AUROC to make the point concrete.
Why not just withhold the image (MMStar's blind test)? It catches leakage through the text side only. On MMMU, GeminiPro scores 42.9% with no image, per Chen et al., which shows how much the text side carries. A blind test cannot see a memorized image–answer link, and our contamination injected only as images is built to evade it.
Proposed Experiments
The controlled experiment follows Park et al.'s setup. We take LLaVA-v1.5-7B and Qwen2-VL-7B and fine-tune them with deliberately injected benchmark items from MMStar, RealWorldQA, NaturalBench and ScienceQA. Injection happens at two levels from the contamination survey's taxonomy: label level (full question, image and answer) and image–answer only (no question text). Exposure counts range from one to several epochs. Each fine-tuned model is matched with a control fine-tuned on benchmark-free data, so contamination labels are known per item. We report instance- and dataset-level AUROC, false-positive rate at a fixed detection rate, the fraction of items eligible for siblings, and the human-audited answer validity of siblings, with a target above 90% before trusting the scores.
The decisive check is three-way. First, on eligible items the full method must beat the diffusion baseline on false-positive rate for clean models. Second, it must beat raw retrieval, showing the filter matters. Third, it must beat augmentation, showing that changing the instance matters beyond changing the pixels.
A second check targets image-only injection, where MM-DETECT and the blind test should fail and our score should not. We then audit released models on MMStar and ScienceQA, where Song et al. report severe leakage for Claude-3.5-Sonnet on ScienceQA. We check whether our item-level flags agree with their text-side flags or reveal contamination that exists only on the image side.
The main risk is coverage. Charts, diagrams and counting questions have no answer-preserving siblings. We will report what fraction of each benchmark is auditable. If that fraction is small on reasoning-heavy benchmarks such as MMStar's mathematics axis, the method's claim narrows to perception-oriented items.
References
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLMs
- Contamination Detection for VLMs using Multi-Modal Semantic Perturbation
- Are We on the Right Way for Evaluating Large Vision-Language Models?
- Google Landmarks Dataset v2 - A Large-Scale Benchmark for Instance-Level Recognition and Retrieval
- On-the-fly learning for visual search of large-scale image and video datasets
- Learning Everything about Anything: Webly-Supervised Visual Concept Learning
- Is Large-scale Pretraining the Secret to Good Domain Generalization?
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models
- Benchmark Data Contamination of Large Language Models: A Survey