Does Contamination Leak Down the Tree of Life? Auditing Few-Shot Fine-Grained Recognition in Multimodal LLMs

Multimodal large language models (MLLMs), which answer questions about images in natural language, are now reported to tell apart look-alike bird species or dog breeds from only four labeled photos per class. Some are also reported to recognize categories they were never shown. These benchmarks, however, have been public for more than a decade. Their images, labels and species names are spread across the web text these models train on. So a plain question has no answer yet: when a model gets a Least Flycatcher right, is it reasoning from what it sees, or recalling that exact photo? And when it gets an "unseen" species right, is that generalization, or leakage from sibling species whose benchmark photos it did see?

We propose to measure this directly. We will inject benchmark image-label pairs into training according to the taxonomy, for example contaminating some species within a genus and leaving their siblings and other genera clean. We then track three quantities through a full few-shot training pipeline that ends in reinforcement learning. The first is how much accuracy is inflated on the contaminated species. The second is how much of that inflation spreads to uncontaminated sibling species. The third is whether the final reinforcement-learning stage hides the contamination from standard detectors while keeping the inflated score. The detection tool is deliberately simple and behavioral. For each species, we compare accuracy on the original benchmark photos with accuracy on freshly collected photos of the same species, and we correct that gap for the difficulty shift between the two photo sets.

Core Claim

In fine-grained recognition, contamination is not a property of isolated test items. It follows the class hierarchy. We hypothesize that exposure to benchmark photos of some species in a genus raises accuracy on held-out sibling species more than on distant species. If so, "unseen category" evaluations that split classes at random, without regard to the taxonomy, mix leakage with generalization. We further hypothesize that the log-probability signature of this contamination disappears after policy optimization, as it does for math. A paired, shift-corrected fresh-image test should still expose it, because that test measures the inflated behavior itself rather than the model's confidence.

The work that motivates this is Fine-R1. It trains Qwen2.5-VL-7B with four-step chain-of-thought rationales, meaning written step-by-step reasoning before the answer. It then applies Triplet Augmented Policy Optimization (TAPO), a reinforcement-learning stage that rewards separating an image from visually similar negatives, using four shots per class on CUB-200, Stanford Cars, Stanford Dogs, Oxford Pets, Flower-102 and FGVC-Aircraft. It reports a +23.75% gain over its base model on unseen categories and never checks for pre-training exposure.

The work that constrains our design is On The Fragility of Benchmark Contamination Detection in Reasoning Models. On math benchmarks, it shows that one epoch of clean GRPO reinforcement learning pushes reference-free detectors toward chance. It also shows that contamination through chain-of-thought data makes models confident on unseen look-alike problems. That paper explicitly leaves open whether the look-alike effect follows a taxonomy. Log-probability detectors are therefore the wrong primary instrument here, and "look-alike" is precisely the variable we control.

The source of the taxonomic splitting logic is BREEDS. BREEDS splits the subclasses of a superclass into disjoint training and evaluation groups and documents 30–40% accuracy drops for supervised CNNs. It never asks whether a web-pretrained model saw the target subpopulations. We reuse its splitting logic and ask that question.

Research Program

The central experiment is a factorial injection study on CUB-200, where species names already group into genera and families (flycatchers, warblers, gulls, and so on). A second replication runs on Stanford Dogs and Oxford Pets, grouped by breed family.

Genera are randomly assigned to three arms. In the label-contaminated arm, benchmark test images of about half the species in each genus are injected with their labels into the chain-of-thought fine-tuning data. This is Level-4 contamination in the taxonomy of the BDC survey and DCR. The remaining species in those genera are held out as "unseen." In the text-only arm, only species descriptions and names are injected (Levels 1–2), with no benchmark images. In the clean arm, nothing is injected.

Every arm passes through the same Fine-R1 recipe: four-shot chain-of-thought fine-tuning, then TAPO on clean data. We take measurements after each stage. This separates what fine-tuning plants from what reinforcement learning preserves or hides.

The measurement instrument needs a second set of photos. For each species, we collect fresh, expert-verified photographs dated after the base model's release. We match them to the benchmark photos on crop, pose and image quality using an off-the-shelf feature matcher, followed by manual review. Let $a_B(s)$ and $a_F(s)$ be a model's accuracy on the benchmark and fresh photos of species $s$. Some of their difference reflects only that one photo set is easier. To estimate that part, we train a reference classifier from scratch on the CUB training split only. It has provably never seen either test set, so its gap $g_{\text{ref}}(s)$ measures pure difficulty shift. The memorization score is

$$ M(s) = \big(a_B(s) - a_F(s)\big) - g_{\text{ref}}(s), $$

and the spillover to an uncontaminated species $s$ in genus $G$ is the difference, in fresh-photo accuracy, between the label-contaminated and clean arms:

$$ S(s) = a_F^{\text{contam}}(s) - a_F^{\text{clean}}(s), \quad s \in G,\ s \text{ not injected}. $$

We compare $S$ for sibling species against $S$ for species in distant families that are also uninjected. Taxonomic spillover means sibling $S$ is reliably larger than distant $S$.

The same instrument then serves as an audit of released models with no injection at all. We compute $M(s)$ for off-the-shelf Qwen2.5-VL-7B, for SigLIP-L, and for a Fine-R1-style model trained on the clean arm. This estimates how much pre-existing web contamination sits inside today's reported fine-grained numbers.

Finally, we propose a taxonomy-aware unseen split, in which held-out species come from genera with no seen member. We report how much of the unseen-category gain survives when evaluation moves from a random split to this split.

Baselines

The naive reference point is what the field does now. It reports top-1 accuracy on the original test photos and "unseen" accuracy on a class split drawn without regard to the taxonomy, with no contamination check. This is the protocol behind Fine-R1's reported gains: +8.51% closed-world over Qwen2.5-VL-7B, +4.27% over SigLIP-L, and +23.75% on unseen categories, all aggregated over six datasets. The file gives no per-dataset absolute accuracies. We will reproduce these numbers under our splits as the starting point. The proposal beats this baseline if it shows that a measurable share of the unseen gain disappears under the taxonomy-aware split or is predicted by sibling exposure.

The first detection baselines are log-probability detectors: Loss, Min-K%, and LiRA with a reference model, all as used in the fragility study. On math, LiRA reached 89.1% AUROC (area under the ROC curve, where 50% is chance) after fine-tuning. It lost about 14 points after one epoch of GRPO, and ten detectors averaged 50–58% AUROC under chain-of-thought contamination. No reported result was found on Lacuna for these detectors on vision-language fine-grained recognition. We run them on answer-token log-probabilities with image and prompt fixed, using the same injected checkpoints.

The strongest existing method, and the one we most need to beat, is the Kernel Divergence Score from How Contaminated Is Your Benchmark?. It measures how much the similarity structure of sample embeddings shifts after a brief LoRA fine-tuning pass, where LoRA is a cheap low-rank weight update. It reports an average Spearman correlation of 0.99 with true contamination rate on text membership benchmarks, against 0.96 for perplexity and 0.32 for zlib. No reported result was found on Lacuna for it on images or MLLMs. We adapt it to late-layer multimodal embeddings with the LoRA budget the paper specifies, following its finding that later layers are most sensitive.

We also include the DCR factor. It fuzzily combines probes at the four contamination levels into a single accuracy discount, and its prompts can be adapted to species names and descriptions.

Every detector is scored in two ways. The first is per-genus AUROC at separating injected from uninjected species. The second is Spearman correlation with the injected fraction, which we vary across 0%, 25%, 50% and 75% of species per genus. Our score $M$ counts as better only if it keeps high AUROC after TAPO while the log-probability detectors and KDS fall. It must also match KDS before reinforcement learning, and it must do so black-box.

Two ablations isolate the key ingredients. We run $M$ without the $g_{\text{ref}}$ correction, to show how much of a raw benchmark-versus-fresh gap is simply a difficulty shift. We also compare random-split with taxonomy-aware unseen evaluation on the same checkpoints, to show that the taxonomy, rather than class count, drives the spillover estimate.

Why not just…?

Why not just collect fresh photos and re-evaluate? This is a real threat, and we adopt it rather than argue with it. The fresh-photo comparison is our instrument, and it extends the chronological comparisons of Chi et al. and Huang et al. down to matched, per-species pairs. On its own, however, a fresh test set cannot say whether a gain on an uncontaminated species comes from its contaminated siblings. It also cannot say whether reinforcement learning erased the internal trace of contamination, and it confounds memorization with difficulty shift. Our factorial injection, the $g_{\text{ref}}$ correction, and stage-wise measurements address these three gaps. The no-correction ablation will show how misleading the naive version is.

Why not just run perplexity, LiRA or KDS on the benchmark? They need either log-probabilities or white-box LoRA access. On math, the fragility study shows the log-probability signal collapses toward chance after GRPO and under chain-of-thought contamination. TAPO is a policy-optimization method of the same kind. Our experiment tests directly whether that collapse transfers to vision. If KDS survives TAPO on images, we will report that honestly as a positive result for KDS, with the caveat that it still cannot audit closed models.

Why not just perturb the benchmark images, with crops, color changes or new backgrounds? The Emperor's New Clothes in Benchmarking? tested 20 mitigation strategies. It found that surface edits keep fidelity, meaning agreement with results on the original benchmark, above 0.9 but give almost zero contamination resistance. Aggressive rewrites dropped MMLU fidelity to 0.686. We include an augmented-benchmark condition to test whether injected models map augmented photos back to their memorized labels, which would reproduce that failure in vision.

Why not just deduplicate the pretraining data? The corpora behind Qwen2.5-VL and SigLIP are not public, so exact-match removal is impossible for the models people actually evaluate. Deduplication would also leave spillover from sibling exposure untouched, since the sibling species are never in the training data.

Decisive Checks

The study is informative whatever the outcome. The claim of taxonomic spillover fails if sibling $S$ is statistically indistinguishable from distant-family $S$ across the injection fractions. In that case, random unseen splits are exonerated, and the decomposition itself is the result. The cloaking claim fails if Loss, Min-K% and LiRA AUROC on injected CUB species stays near its pre-reinforcement-learning level after TAPO. The instrument fails if $M$ does not rise monotonically with the injected fraction, or if the $g_{\text{ref}}$ correction does not bring clean-arm $M$ to about zero.

The audit claim matters most in practice. It succeeds if off-the-shelf Qwen2.5-VL-7B shows a positive corrected $M$ on CUB and Stanford Dogs, and if the unseen-category gain of the Fine-R1-style model shrinks under the taxonomy-aware split. Either way, the field would obtain a split protocol and a black-box memorization score for fine-grained benchmarks, and both can be reused whenever a new few-shot result is reported.