Which Path Earned the Point? Separating Perception, Language Priors, and Memorization in Vision-Language Benchmark Scores

When a vision-language model (a language model given access to images) answers a benchmark question correctly, at least three different things could have produced that answer. The model may have looked at the image and reasoned. It may have guessed from the wording of the question and options, which is a linguistic shortcut. Or it may have seen this exact test item during training and recalled the answer, which is contamination. Current tools each test for one of these in isolation, and they interfere with each other. A contamination detector that edits the image cannot see memorization on items the model can already answer from text alone. A text-only filter cannot see memorized images. Neither can tell a model that really perceives from one that has simply learned that synthetic images are easier.

We propose to treat each benchmark item as a small causal system whose variables the evaluator can intervene on. The evaluator removes the image, edits its meaning, regenerates its pixels while keeping its meaning, paraphrases the text, and swaps the distractor options. The goal is to estimate, item by item, how much of a model's correctness flows through perception, through language priors, and through memory of that specific item. The causal-inference literature supplies two things here. It tells us which combinations of interventions are sufficient to identify each path. When a needed intervention cannot be run, for example because the image generator cannot faithfully edit geometry diagrams, it gives bounds rather than guesses.

Core Claim

The headline result of the best existing VLM contamination detector, image-level semantic perturbation, mixes two effects, and the mixing can be undone with one extra intervention. Contamination Detection for VLMs Using Multi-Modal Semantic Perturbations edits an image with a diffusion model so that a different option becomes correct. It reports that clean models improve by +31.5% on MMStar while contaminated models fall by as much as −45%. That positive shift for clean models is itself an artifact: the authors attribute it to synthetic images isolating the salient cue. The raw score difference therefore measures memorization loss minus synthetic ease, and the two cannot be separated from that arm alone.

A semantics-preserving regeneration solves this. The same pipeline is used to redraw the image so that the original answer stays correct. This arm changes the pixels the model might have memorized while holding the meaning fixed. It therefore identifies synthetic ease separately from memorization. Adding an image-removed arm and a text-surface arm (paraphrase, option shuffle, distractor replacement) extends identification to the items the perturbation paper explicitly gives up on: those answerable from language alone. The new thing is an intervention design with stated identification conditions. It yields three per-item attributions (perception, prior, memorization), checked against models into which we deliberately inject known contamination and known shortcuts.

Research Program

The causal model is deliberately coarse. We cannot write down the micro-level graph inside a vision-language model or inside an image. Each item is therefore described by clusters of variables, in the sense of Causal Effect Identification in Cluster DAGs. That paper shows that identification algorithms run soundly on a graph over groups of variables whose internal structure is left unspecified, provided the cluster graph is acyclic and treatment and confounder are not merged into one cluster. The paper supplies our method, and its warning about merging constrains our cluster choices.

Our clusters are as follows. The first is item identity, meaning whether the item was in training. The second is visual surface, the pixel-level configuration. The third is visual semantics, the scene content that determines the answer. The fourth is text surface, the exact wording and option order and identity. The fifth is the language prior, the plausibility of each option given only the text. Item identity influences both surfaces. Visual semantics determines the gold answer. The model's response can depend on all five.

The difficulty a micro-graph would hide is that item identity confounds both surfaces with the answer. A memorizing model looks like a perceiving model unless surface and semantics are moved independently. That is exactly why separate "surface-only" and "semantics-only" interventions are required. Visual surface and visual semantics must be kept as separate clusters. Merging them is precisely the granularity error the cluster-graph paper warns destroys identifiability, and it is what the single-arm perturbation detector implicitly does.

For each item and model, we run six conditions. The first is the original item. The second removes the image. The third edits the image's semantics so another option is correct. The fourth regenerates the image with semantics preserved. The fifth applies the semantics-preserving regeneration together with a paraphrase and fresh distractors. The sixth applies the semantic edit with a fresh text surface.

From these we define three counterfactual quantities per item, each written as a probability over the model's sampled responses. The memorization attribution is the drop in correctness from the original item to the condition with surfaces regenerated and semantics preserved. This drop is corrected by the same drop measured on certified-clean reference models, which estimates synthetic ease. The language-prior attribution is the probability that the model gives the original answer after the semantic edit or with the image removed. The perception attribution is the probability that the model follows the semantic edit to the new correct answer under a fresh text surface.

Under the cluster graph, all three are identified from these interventions. For items where a condition cannot be produced faithfully, we use the bounding approach of Partial Counterfactual Identification from Observational and Experimental Data. Faithful generation fails on fine text and precise geometry, which the perturbation paper names as a bottleneck. That paper computes bounds on counterfactual probabilities from the available observational and interventional distributions. Our graph has a handful of discrete clusters, so its known blow-up with large parent sets does not bite.

We contribute four things. The first is the intervention generator, which extends the Flux-plus-edge-guidance pipeline with the semantics-preserving and text-surface arms. The second is an identification analysis stating which attributions each subset of arms identifies. The third is the per-item estimator with bounds. The fourth, and the part that makes the work checkable, is a validation suite of models with planted causes.

Baselines

The naive reference point is raw accuracy on the original benchmark, which credits every correct answer to perception. Beside it we place MMStar's Multi-modal Leakage metric from Are We on the Right Way for Evaluating Large Vision-Language Models?. That metric compares a VLM without images against its own language backbone without images, and a positive gap signals leakage. We also use the Linguistic Gap from The Role of Linguistic Priors in Measuring Compositional Generalization of Vision-Language Models. It contrasts overall accuracy with accuracy on items where GPT-2 prefers a distractor, and it exposed NegCLIP's gap of 13.24 versus CLIP's 3.43. These are the standard aggregate diagnostics for leakage and shortcuts respectively. All of them are computed on exactly the same items and models we use.

The strongest competitor is the single-arm semantic perturbation detector itself, which we rerun with its published pipeline and filtering. In its own setting it reports that clean LLaVA-v1.5-7B scores about 38% on MMStar and about 69% on the perturbed version. After three epochs of contamination, the original score rises to about 54% while the perturbed score falls to about 38%. We will reproduce this contaminated model with the same three-epoch protocol.

The comparison that matters is item-level and model-level discrimination of planted contamination. Our measure is the area under the curve for separating contaminated from clean (model, item) pairs. For this measure, no reported result found on Lacuna for semantic perturbation, so we compute it ourselves for both methods. Our method wins if two conditions hold. Its discrimination must exceed the single-arm detector on the full benchmark. It must also be clearly above chance on the subset of items that text-only models can answer, where the single-arm detector should be near chance by its authors' own admission.

The key ablation removes the semantics-preserving arm and nothing else, using the same generator, compute, and number of samples per item. If memorization estimates then collapse back into the single-arm score difference, that isolates the contribution. A second ablation removes the text-surface arm to show what is lost on shortcut-answerable items. All methods receive the same diffusion compute and the same ambiguity filter per item. No method is tuned on the planted-cause models used for scoring.

Why not just…?

Why not just filter out text-answerable items first, then run image perturbation on the rest? This is the most serious alternative. On an already-filtered benchmark like MMStar it may do nearly as well, and we will test exactly that. It falls short in three ways that our experiments measure. First, filtering throws items away. On shortcut-heavy sets such as ARO, the text-only GPT-2 reached 85.07% on the Visual Genome relation task. The filter would leave little, and it says nothing about the discarded items, whose scores still appear on leaderboards. Second, the filter uses other models' priors, not the evaluated model's. Third, filtering does nothing about synthetic ease, which still mixes into the score difference. If the filter-then-perturb pipeline matches us on retained items, that is a useful finding. Our claim would then narrow to coverage and calibration rather than detection power.

Why not just report Multi-modal Leakage and the perturbation score difference side by side? The two numbers answer different questions on different scales, and neither corrects for synthetic ease. Multi-modal Leakage needs access to the base language model and is an aggregate. It cannot say which items, and it cannot detect memorization that is tied to images rather than text. The perturbation paper shows that text-only perturbations miss this form of memorization. Our planted-cause grid tests this directly: in a model with both injected shortcuts and contamination, the side-by-side readout should mislabel items that our joint design labels correctly.

Why not just build a fresh, procedurally generated benchmark instead of diagnosing old ones? DyVal shows that generating test items from random graphs defeats static contamination, which is why phi-1.5 and WizardMath-13B drop to near zero on DyVal tasks. But it is limited to symbolic tasks. Fresh items still carry language priors, so freshness removes only one of the three paths. Fresh data also leaves the scores the field already reports uninterpretable. Our generator does produce fresh surfaces, so it inherits DyVal's benefit at the item level.

Why not just use membership inference such as Min-K% Prob? Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? uses it to certify that text is fresh. We found no evidence in the file that it detects memorization tied to images in VLMs. It also needs token probabilities, which closed models often do not expose. We include it as an extra baseline on the open models.

Proposed Experiments

The decisive experiment is a two-by-two planted-cause grid on LLaVA-v1.5-7B and Qwen2-VL, the models used by the perturbation paper. One factor is contamination: three-epoch fine-tuning on half of the MMStar, NaturalBench, and RealWorldQA test items, with the other half held out. The other factor is shortcut injection: fine-tuning on a VQA v2 resample in which the co-occurrence rules mined in Beyond Question-Based Biases always hold, evaluated against its counterexample split. On that split VilBERT already falls from 80.50% to 39.24%. Because we know which items were leaked and which shortcuts were planted, each attribution has ground truth.

The success criteria are as follows. Memorization attributions should separate leaked from held-out items. Language-prior attributions should rise for shortcut-injected models specifically on items where the planted rule fires. Perception attributions should stay stable across both manipulations for items with neither cause.

The second check targets synthetic ease. On clean models, the semantics-preserving arm should reproduce most of the positive shift that the perturbation paper reports. Correcting for it should move clean models' memorization estimate to near zero.

The third check covers generation failures. On the MathVerse testmini subset, where diffusion editing of diagrams is unreliable, we report bounds rather than point estimates. The bounds pass if they contain the planted ground truth and are narrow enough to rank models. MathVerse's finding that removing the diagram can raise accuracy by 5% or more is a natural stress case for the image-removed arm.

Finally, we apply the method to unmodified public models on MMStar, NaturalBench, and MMMU-Pro. For each model we publish the decomposition of every reported accuracy into perception, prior, and memorization shares. We also flag where the shares disagree with Multi-modal Leakage and the Linguistic Gap.