When a vision-language model solves a benchmark problem about a chart or a geometry figure, we usually count only whether its final answer matches the key. The model might have read the figure and reasoned from it. It might instead be answering from the question text alone, or from a memory of that exact problem picked up in pre-training. This note proposes a simple behavioral test that separates these cases. Give the model the original question paired with a deliberately wrong picture: a minimally edited image whose correct answer is different. Then check whether it still returns the original key. A model that reasons from the image should change its answer. A model that has memorized the item, or that ignores the image, will not. We call the rate at which the original answer survives the edit gold-answer persistence. The central claim is that this rate detects benchmark contamination in multimodal models, and that it keeps working after reinforcement-learning fine-tuning. That setting is exactly where the standard detectors, which rely on token probabilities, are known to fail.
The idea borrows its vocabulary from noisy correspondence, the problem of web image-text pairs where each item is valid but the link between them is broken. Here we break the link on purpose, as a probe. The opportunity matters now for two reasons. The literature shows that outcome-only scoring and outcome-only rewards can hide both ungrounded reasoning and memorization. Separately, multimodal contamination auditing has no accepted diagnostic: existing leakage tests such as partial-prompt completion are text-only.
Core Claim
Our claim is that deliberately mismatched image-question pairs give a contamination signal that does not depend on the model's loss or token probabilities. Because the signal is behavioral, we expect it to survive the reinforcement-learning step that erases loss-based evidence. The same measurement also yields a corrected score, grounded accuracy: the model counts as correct on an item only if it is right on the original image and also right on its counterfactual twin. Grounded accuracy is a direct fix for outcome-based bias on visual math and chart benchmarks.
Formally, consider a benchmark item with image $I$, question $q$ and key $a$, together with counterfactual images $I'_1,\dots,I'_m$ whose keys $a'_j$ all differ from $a$. Let $f$ be the model's answer function. We define persistence for that item as
$$ P(q) = \frac{1}{m}\sum_{j=1}^{m} \mathbb{1}\left[f(I'_j, q) = a\right]. $$
Raw persistence mixes two causes. One is memorization. The other is ordinary "textual inertia," the tendency for long written reasoning to drown out attention to the image. To separate them, the contamination score for an item is its persistence minus the persistence of an independent reference model on the same item. This follows the reference-model logic of likelihood-ratio membership attacks, but applies it to behavior instead of loss.
Motivation and Prior Work
Several papers establish the gap this proposal targets.
The perception-reasoning literature shows that models often do not use the image even when they succeed. MATH-Vision curates 3,040 competition problems and attributes 42.2% of GPT-4V's errors to reasoning and 31.9% to recognition. Plot Twist probes charts with elementary questions and finds that ChartQA-style success leans on chart titles: accuracy is up to 79% when the title contains the answer and falls to the 30% range without it. Are VLMs Seeing or Just Saying? swaps the image for a subtly altered one mid-generation. ERNIE-4.5-VL-Thinking falls from 79.9% to 19.6%, and "thinking" models fall further than instruct models.
These papers motivate the probe but do not use it for contamination. They diagnose grounding failure; none asks whether answers that survive the swap are memorized. Our probe also differs mechanically from the mid-generation swap: we swap at the input. A model that honestly reads its input has every chance to follow the new image, so a surviving gold answer is more telling.
The contamination literature supplies the constraint that shapes the design. Reasoning or Memorization? shows that Qwen2.5 reaches 65.8% exact match on MATH-500 from only an 80% prefix of the problem. It also shows that "gains" from random-reward reinforcement learning vanish on the clean RandomCalculation generator. On The Fragility of Benchmark Contamination Detection in Reasoning Models is the paper this proposal most directly answers. It shows that brief GRPO on clean data drives loss-based detectors such as LiRA and Min-K% toward chance. It traces the cause to policy-ratio clipping, which shrinks the probability gap between seen and unseen items. After contamination with chain-of-thought traces, all ten detectors it tests sit at 50–58% AUROC.
Both papers are text-only. The second leaves open a detector that survives this "cloaking" by reinforcement learning. Our bet is that clipping compresses probability differences but cannot make a memorized answer disappear when the evidence for it is removed. That bet is testable, and it could fail if chain-of-thought contamination teaches transferable skill rather than item answers. The fragility paper suggests that this happens in its second stage, and we treat that case explicitly below.
On the outcome-bias side, Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning shows that outcome-only rewards can leave the final answer unchanged when reasoning is truncated. Its causal-importance metric asks whether the answer depends on the reasoning tokens. Our persistence asks whether it depends on the image, which is the multimodal analogue that paper does not cover. URSA uses step-level process rewards to penalize correct answers reached through flawed visual steps. Its reported MathVision score of 31.5% against GPT-4o's 30.4% is exactly the kind of margin that has not been audited for leakage.
From the noisy-correspondence literature we reuse one technique. Geometrical Structure Consistency Learning fits a two-component Gaussian mixture to per-pair scores to separate clean pairs from noisy ones without labels. We fit the same kind of mixture to item-level persistence scores to separate "tracks the image" items from "ignores the image" items without knowing which items leaked. That line of work treats broken correspondence as something to repair during training; we create it on purpose at evaluation time.
Research Program
The work has three parts.
The first is a counterfactual pair set. For charts and function plots we re-render from the underlying data with one value or parameter changed, so the new key is computable. For the math benchmarks we start from the 800 altered image pairs in VS-BENCH, drawn from MathVista, MathVerse, MathVision and MMMU-Pro. We first verify whether keys for the altered images are provided. Where they are not, we have annotators re-solve the edited item and keep only pairs with a changed, unambiguous answer. We report the size of the retained set rather than assume it.
The second is a controlled-contamination testbed that gives ground-truth membership. We randomly split the evaluation items into a member half and a non-member half. We then contaminate Qwen2.5-VL-7B on the members in two ways, mirroring the two stages of the fragility study. In the first, we supervise it on member images, questions and answers, then run a brief GRPO phase on clean multimodal math data disjoint from the benchmark. In the second, we supervise it on member chain-of-thought traces instead. We also vary contamination form. Text-only leakage (question and answer without image) and image-and-text leakage should behave differently under a wrong-picture test, and that difference is itself informative about how multimodal memorization is stored.
The third is an in-the-wild audit of released open models on MathVista and MATH-Vision. For each model we report raw accuracy, grounded accuracy, and the mixture-model estimate of the fraction of items answered without using the image. We compare these against a continuously refreshed reference such as MMBench-Live and freshly re-rendered items.
A secondary analysis asks whether training signal matters. In the testbed, we compare outcome-reward GRPO against a URSA-style process-reward model and measure whether process supervision lowers persistence on member items. The file reports no such comparison, so this is an open question rather than an assumption.
Baselines
The naive reference point is the do-nothing audit: report raw benchmark accuracy and assume it is clean. In the testbed this "detector" has AUROC 0.5 by construction. The member-minus-non-member accuracy gap shows how much inflation it misses.
The strongest existing detector family is loss- and likelihood-based membership inference: Min-K%, plain loss, and LiRA with reference models, all applied to the answer tokens conditioned on image and question. These come from the tool set evaluated in the fragility paper, which reports LiRA at 89.1% AUROC on text reasoning models before reinforcement learning, dropping by about 14 points after a brief GRPO phase. After chain-of-thought contamination, it reports all ten detectors at 50–58% AUROC. No reported result was found on Lacuna for any of these detectors on a vision-language model. We will therefore run all of them ourselves on the same checkpoints, with the same reference model and the same member split. Beating them means persistence AUROC stays substantially above chance after GRPO, in the regime where the likelihood detectors fall near 0.5. Merely matching them before reinforcement learning does not count as beating them.
The strongest behavioral competitor is partial-prompt completion from Reasoning or Memorization?. On text it gave 65.8% exact match from 80% prefixes. We run it with the image present and with the image absent, at matched query budgets.
The blind baseline asks the question with no image, or a blank one. It is the key ablation because it isolates what the counterfactual image adds over simply removing the image.
A second ablation replaces the counterfactual with a random unrelated image. This separates "wrong but plausible evidence" from "irrelevant evidence."
All methods receive the same number of model queries per item and the same reference model. Hyperparameters such as the Min-K fraction are tuned on a held-out split, not on the test members.
Why not just…?
Why not just ask the question without the image? Blind accuracy measures reliance on text priors, but it confounds memorization with legitimately guessable questions. MathVista reports that a frequent-guess heuristic reaches 26.3%. Models may also refuse or hedge without an image, which deflates the signal. A counterfactual image keeps the input well-formed and forces a different correct answer, so a surviving gold answer cannot be explained by guessing the common option. This is a real threat. If blind persistence matches counterfactual persistence in AUROC within noise, we will report that the cheaper test suffices and drop the pair construction.
Why not just use fresh or dynamic benchmarks? Resources like MMBench-Live and PuzzleBench avoid contamination going forward. They cannot audit claims already made on MathVista or MathVision, such as URSA's margin over GPT-4o, and they do not say which items of a legacy benchmark are compromised. Re-rendered counterfactuals are in fact a way to refresh legacy benchmarks item by item. We use fresh benchmarks as the external check on our estimates rather than as a competitor.
Why not just extend loss-based detectors to image-conditioned likelihoods? The fragility paper gives a mechanism for why these fail after GRPO: clipping compresses the loss gap. That mechanism does not depend on modality, so we expect the same collapse. We still run the detectors as the main baseline, so this is settled empirically rather than assumed.
Why not just grade the reasoning with a perception-aware judge? Perception-Judge-Flex-7B raises perceptual-error detection from 69.5% to 85.7% by training a judge on perturbed responses. A memorized item, however, typically comes with a correct, fluent, visually consistent rationale, so there is nothing for the judge to flag. We include the judge in the testbed to measure how often it rates member-item responses as grounded. We expect it to do so often, which would be the evidence against this alternative.
Proposed Experiments and Decisive Checks
The decisive experiment is the post-GRPO testbed comparison. Contamination should raise member accuracy. The likelihood detectors should fall toward chance after the clean GRPO phase, as the fragility paper found in text. If persistence minus reference-persistence still separates members from non-members well above chance, the core claim holds.
The honest failure case is the second stage, chain-of-thought contamination. If the model learns transferable procedures rather than item answers, persistence may drop to non-member levels. The fragility paper would then be read as saying there is no item-level memory left to detect, and we would report that result as such.
A second check guards against confounding. On uncontaminated reference models, persistence should correlate with known grounding weakness, for example higher for thinking variants than for instruct variants, consistent with the VisualSwap findings. The difference score should not show that pattern. If the difference score still tracks model type rather than membership, the reference calibration is inadequate.
A third check concerns grounded accuracy. On released models, grounded accuracy should shrink the reported gaps between models that differ mainly in training data exposure. It should also agree more closely with fresh-benchmark rankings than raw accuracy does. We measure that agreement by rank correlation.
Finally, the mixture-model estimate of leaked items is validated in the testbed against true membership, reported as precision and recall, before it is applied in the wild.
References
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination
- Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning
- Unlocking Multimodal Mathematical Reasoning via Process Reward Model
- Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- PuzzleBench: A Fully Dynamic Evaluation Framework for Large Multimodal Models on Puzzle Solving
- Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge for Mathematical and Scientific Reasoning Evaluation