When a vision-language model scores well on a benchmark, part of that score may come from having seen near-copies of the test items during training. Today the field deals with this using hard cutoffs. A question is dropped if more than two of eight text-only models can answer it without the image. A training item counts as a duplicate if its embedding similarity to a test item exceeds some fixed number. These cutoffs are chosen by hand. Work on threshold sensitivity in other fields shows that conclusions drawn from such cutoffs often reflect the cutoff more than the underlying data. We propose to replace the keep-or-drop filter with a price. For each benchmark item, we measure how close it is, as an image–text pair, to the model's training data. We then subtract the accuracy boost that this degree of closeness is known to produce. That known boost comes from a dose-response curve fitted on deliberately contaminated models.
The proposal combines three ingredients. Frozen pre-trained encoders supply cheap, fixed embeddings that make similarity search possible. The image and text embeddings are fused early, before any similarity is computed, so that a leaked pair is recognized as a pair even when its text has been paraphrased and its image re-cropped. The output is a contamination-adjusted accuracy that changes smoothly with the evidence instead of jumping at an arbitrary boundary. The test of success is whether the adjusted score predicts how the model does on items it truly never saw, and whether model rankings stop flipping when the analyst's settings change.
Core Claim
Our central claim is that multimodal contamination should be measured on the fused pair and corrected by its expected effect, not by a binary filter.
For a benchmark item $i$, let $s_i$ be its highest fused similarity to any training pair. Let $c_i \in \{0,1\}$ record whether the model answered it correctly. Let $g(s)$ be the expected accuracy gain from training exposure at similarity $s$, which is the dose-response curve. We define the adjusted accuracy as
$$\hat{A} = \frac{1}{N}\sum_{i=1}^{N}\big(c_i - g(s_i)\big),$$
where $N$ is the number of items. We fit $g$ on models whose leakage we control. Because $g$ is smooth, a small error in $s_i$ moves $\hat{A}$ by a small, bounded amount. A hard filter behaves differently: when an item crosses the cutoff, it enters or leaves the denominator entirely, and the score jumps.
The novelty is not soft weighting in itself. It is the combination of three things: a pair-level similarity score built from frozen, early-fused encoders; an inflation curve fitted empirically to that score; and a stability guarantee that hard-cutoff decontamination cannot offer.
Why this gap is real
Several papers motivate the proposal by documenting a gap.
Soft Contamination Means Benchmarks Test Shallow Generalization uses cosine similarity over frozen text embeddings to find paraphrased duplicates in the Olmo3 training data. It shows that these duplicates inflate scores by about 15–20%. However, it flags candidates with a hard similarity cutoff and studies only text and code.
Are We on the Right Way for Evaluating Large Vision-Language Models? introduces two measures. Multimodal Gain is the accuracy with the image minus the accuracy without it. Multimodal Leakage asks whether a vision-language model answering without images beats its base language model. The paper builds MMStar using the "more than 2 of 8" rule and reports that text alone reaches 42.9% on MMMU. It never asks how sensitive its rule is.
Both Text and Images Leaked! introduces MM-DETECT, which detects contamination from how much a model's outputs degrade when options are shuffled or captions perturbed. It finds that much of the contamination is inherited from the underlying language model. Its verdicts are behavioral and fall into coarse severity bins, and it never looks at item representations.
All three papers treat contamination as present or absent at the item level. Our proposal treats it as a quantity with an effect size.
Other work supplies the motivating lens on thresholds. Bozhilova et al. show that rankings of hub nodes in a network reorder under small threshold changes. Janssen et al. show that graph metrics on binarized brain networks track the researcher's chosen threshold rather than the biology. Geurts et al. and Benard et al. show that exact split points in decision trees produce jagged, high-variance outputs. We transfer this diagnosis to leaderboards: the rankings of models are the "hubs" that may be cutoff artifacts.
PaCoST constrains the design. It removes hand-tuned thresholds by running a paired t-test on model confidence over original versus rephrased prompts. This still ends in a p < 0.05 decision, and it relies on token probabilities. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that such probabilities can be erased by reinforcement-learning fine-tuning: detector AUROC falls from 89.1% to 50–58%. This is the reason we score data similarity rather than model confidence. Similarity between test items and training data does not depend on the model's output probabilities, so post-training cannot hide it. One caveat follows directly. Our method needs the training corpus, so it applies to open-data models and to controlled studies, not to closed models.
The method draws on two further lines of work. On frozen extraction, Lin et al. show that frozen backbones rival full fine-tuning, and Bazaga et al. show that frozen embeddings plus shallow learners do well with little data. Together they justify using fixed encoders with a small trained head. On fusion, Kiela et al. fuse frozen visual and text vectors at the vector level, and Liu et al. show that meme meaning exists only in the image–text interaction. The latter is the reason a pair can be leaked even when neither half looks leaked, and novel even when both halves look familiar.
Research Program
We will build a pair-similarity auditor. It has frozen text and image encoders. Their normalized embeddings are concatenated, and a small projection head is trained contrastively on top. The head learns that a pair and its transformed version (paraphrased text plus an augmented image) belong together. It also learns that "same text, different image" and "same image, different question" decoys do not. The encoders themselves stay frozen.
We will index the training mixture of an open vision-language model and compute $s_i$ for every item in ScienceQA, MMStar, and MMMU.
To fit $g$, we will fine-tune copies of a LLaVA-1.5-class model on that mixture with benchmark items injected under controlled conditions. We will vary three things: the exposure level (from the data level, meaning questions without answers, to the label level, meaning question–answer pairs, following the taxonomy in Benchmark Data Contamination of Large Language Models: A Survey); the transformation applied (exact copy, text paraphrase, image augmentation, both, or mismatched pairing); and the dose. We will fit $g$ as a monotone function of $s$ on half of the injected items and test it on the other half.
A secondary question is whether $g$ itself changes after a GRPO-style reinforcement-learning stage. The fragility paper reports that pass@1 inflation survives such training even when detectors fail, so we expect the curve to persist.
Baselines
The do-nothing reference is raw benchmark accuracy with no adjustment. For context, the file reports 57.1% for GPT-4V on MMStar and a Multimodal Gain of only 5.8% on MMMU (Are We on the Right Way…). We will report raw accuracy for our controlled models on the same splits.
The strongest existing filtering method is MMStar-style blind filtering at the published 2-of-8 rule. We will run it with the same text-only judges, swept over every cutoff from 1 to 7 of 8 and over three prompt templates.
The strongest existing detector is MM-DETECT. We will run its option-shuffle test on ScienceQA and MMStar and its caption slot-guessing test on COCO-Caption2017 and NoCaps, on our controlled models. No reported result was found on Lacuna for MM-DETECT detection accuracy against ground-truth leakage. We therefore report its AUROC on our injected items as the number to beat.
The text-only soft-contamination pipeline from the Soft Contamination paper is the closest unimodal relative. We will run it with the same frozen text encoder and index, with its hard cosine cutoff swept.
Two ablations isolate our key ingredients. The first isolates fusion: it replaces the fused score with late combinations of unimodal similarities, taking the maximum, minimum, or product of the text and image cosines. The second isolates soft pricing: it uses our fused score but applies hard cutoffs instead of $g$.
All methods share the same encoders, index, injected models, and a calibration split with equal tuning budget.
We count our method as better if it does all of the following:
- achieves higher AUROC than MM-DETECT and the unimodal pipeline on transformed-pair leakage;
- produces adjusted accuracy closer to accuracy on truly unseen items than any filter at any cutoff;
- yields higher Kendall rank stability across analyst settings.
Why not just…?
Why not just report results at several cutoffs and call it a sensitivity analysis? This is the most serious threat, and we include it as a baseline. A sweep shows that conclusions are unstable, but it does not say which number is correct. If the hard filter's estimate at its best cutoff, chosen after the fact, matches unseen-item accuracy as well as $\hat{A}$ does, then pricing adds only the stability property. We would report that honestly and recommend the sweep. We expect $g$ to win in the regime that matters most: partially transformed leaks at intermediate similarity, where any single cutoff either misses inflated items or discards clean ones.
Why not just use n-gram matching and perceptual image hashes? The Soft Contamination paper shows that exact matching misses semantic duplicates of 78% of CodeForces problems in Dolma3. Test of Time shows that rephrasing drops the influence hit rate from 77.5% to 17.5%. Our paraphrase-plus-augmentation injections test the multimodal version of this failure directly, and we will include hashing in the detection comparison.
Why not just use Min-K% probability, loss, or LiRA on the model? These detectors fall to 50–58% AUROC after reinforcement learning or distillation (fragility paper), and PaCoST notes that they fail on short multiple-choice items. We will run them on our controlled models before and after the reinforcement-learning stage. If they match our detection accuracy before that stage, the advantage of similarity scoring rests on its robustness after it.
Why not just compute similarity separately for text and image and take the max? This is exactly the late-fusion ablation. The mismatched-pair decoys decide the question. Unimodal maxima should flag "seen text, new image" items as leaked even when the model gains nothing from them. If the late-fusion ablation equals early fusion on both detection and the fit of $g$, then fusion is unnecessary and we drop it.
Decisive Checks
Three experiments decide the proposal.
The first is held-out validity. For each controlled model, we hold back a fresh split that was never injected and is matched in difficulty. We then ask whether $\hat{A}$ on the contaminated split predicts accuracy on the fresh split better than raw accuracy and every filtered score. This is measured by absolute error across all injection conditions.
The second is rank stability. We take public open-data models evaluated on MMStar and MMMU and compute Kendall's tau between leaderboards produced under different encoder choices, cutoffs, and judge sets. Our adjusted score should vary far less than the hard filters.
The third is pair-level detection. Fused similarity should beat late-fusion similarity specifically on transformed pairs and decoys, and should be no worse on exact copies.
If the first check fails, the dose-response curve does not transfer and the proposal reduces to a sensitivity report. If the third check fails, the fusion ingredient is removed and the method becomes unimodal similarity pricing.
References
- Soft Contamination Means Benchmarks Test Shallow Generalization
- Are We on the Right Way for Evaluating Large Vision-Language Models?
- Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLMs
- Measuring rank robustness in scored protein interaction networks
- Quantifying uncertainty in brain network measures using Bayesian connectomics
- Closed-form dual perturb and combine for tree-based models
- Interpretable Random Forests via Rule Extraction
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Could Giant Pre-trained Image Models Extract Universal Representations?
- TabMDA: Tabular Manifold Data Augmentation for Any Classifier using Transformers with In-context Subsetting
- Comparing Data Sources and Architectures for Deep Visual Representation Learning in Semantics
- FigMemes: A Dataset for Figurative Language Identification in Politically-Opinionated Memes
- Benchmark Data Contamination of Large Language Models: A Survey
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination