Few-shot image recognition is now routinely evaluated by freezing a large self-supervised backbone and asking a simple classifier to separate hundreds or thousands of classes from a handful of labeled images per class. Examples include 200-way CUB, 1000-way ImageNet-1K, and 100-way Danish Fungi. Nobody checks whether the backbone already saw the evaluation images during pretraining. Nobody checks whether the preprocessing used at evaluation matches the preprocessing used in pretraining. Both questions matter more as the number of classes grows. Separating two hundred bird species depends on fine, instance-level detail, which is exactly what memorizing specific images would supply. A mismatched resize, crop or compression step can erase that advantage, or hide it.
This proposal measures both effects in a controlled setting. We pretrain backbones ourselves so that we know which evaluation images they saw, and we cross that with deliberately mismatched preprocessing. The goal is to estimate how much of reported many-way accuracy is memorization, and how that share changes with the number of classes and with how semantically close the classes are. We also test a contamination audit for the realistic case where the pretraining corpus is unknown. The audit treats a benchmark as a set of images. It infers one shared summary per benchmark, separating "how familiar these images look to the backbone" from "how this benchmark was preprocessed." It is a global-local latent model in the style of the Neural Statistician. We report it only if it beats much simpler audits.
Core Claim
We make two empirical claims. The first is that the accuracy gain from the backbone having seen evaluation images grows with the number of classes, and grows as classes become semantically closer. The reason is that coarse class features carry easy low-way tasks, while fine-grained many-way tasks rely on detail that memorization provides.
The second claim is that preprocessing mismatch confounds contamination audits. A contaminated benchmark that is resized or compressed differently from pretraining looks less familiar to the backbone, so averaged per-image "was this seen?" scores call it clean. An audit that looks at how familiarity changes as the evaluation pipeline is varied can separate the two factors, because preprocessing shifts all images alike while membership shifts only some.
The first claim stands on its own as a measurement result even if the audit turns out to be unnecessary.
Why this sits where it does
The many-way literature motivates the question. Few-Shot Image Classification Benchmarks are Too Far From Reality shows that uniform class sampling produces easy tasks, with a median coarsity of 85.1 on tieredImageNet. Coarsity is a WordNet distance between the sampled classes. Semantic task sampling costs top models 12 to 15 points, and 100-way 1-shot Danish Fungi (DF20) reaches only 9.75%. Instance-based Max-margin for Practical Few-shot Recognition defines the "practical" 200- to 1000-way setting on frozen DINO, MoCov3 and MSN backbones. Both papers assume the backbone's features are untainted, and neither varies preprocessing. We keep their protocols and their coarsity measure and add the two missing controls.
The contamination literature supplies the concern and constrains the method. Task Contamination uses chronological splits to show that language models' few-shot gains largely vanish on post-cutoff tasks. That is the language analogue of our first claim, but it does not study many-way classification or vision. Rethinking Pretraining Data Detection for LLMs builds a sequence-level detector over token log-probabilities. Its sequence-level signal does not transfer to a frozen image encoder. It also leaves the problem of detecting contamination of an entire benchmark, rather than single items, unsolved.
The preprocessing literature supplies the confounder. The Devil is in the Details finds that 63% of event-extraction papers omit their preprocessing, and that script choices swing F1 by up to 9.2%. Let's Stop Incorrect Comparisons in End-to-end Relation Extraction! shows that the "same" dataset ranges from 10k to 14k sentences across versions. We carry this lesson to image pipelines, where crop policy, interpolation, compression and normalization constants are similarly undocumented.
The audit borrows its mechanism from hierarchical latent models. Towards a Neural Statistician encodes each set member, pools the encodings with a symmetric mean, and infers one context variable per dataset. Unsupervised learning of global factors in deep generative models shows that a batch-level latent can absorb batch-wide traits separately from per-item content. A linear classifier on that latent separated structured batches with 0.95 accuracy. We reuse the set-pooling design. We do not generate images, and we use the global variable as an estimator of benchmark-level nuisance and membership.
Research Program
The controlled backbones come first. We pretrain DINO-style ViT-S/16 encoders on a fixed base corpus: ImageNet-1K training images with a held-out pool removed. We then inject DF20 and CUB images under four exposure conditions. In the first, no evaluation images are seen. In the second, only the images later used as labeled support examples are seen. In the third, both support and query images are seen. In the fourth, the classes are seen through other images but the specific evaluation images are not. Injection fractions range from 0 to 100% of the evaluation pool. The base corpus, compute and schedule are identical across conditions, so any accuracy difference is attributable to exposure. Each condition is pretrained under two documented preprocessing pipelines that differ in crop policy, interpolation and JPEG quality. Each condition is then evaluated under both pipelines, which gives a crossed design of matched and mismatched preprocessing.
Measurement is the second component. On each backbone we run nearest-neighbor, linear-probe and IbM2 classifiers at 5, 20, 100 and 200 ways (1000 for the ImageNet pool), with 1 and 5 shots. Tasks are sampled both uniformly and by coarsity tier, following the semantic task sampling of Few-Shot Image Classification Benchmarks are Too Far From Reality. The central quantity is the contamination gain $\Delta(w,\kappa)$: accuracy with exposure minus accuracy without, as a function of way count $w$ and coarsity $\kappa$. We estimate it with episode-level bootstrap intervals and report it separately for matched and mismatched preprocessing.
The audit is the third component. For each evaluation image we compute a familiarity profile. The profile records how stable the backbone's embedding is when the image is re-processed under a grid of pipeline variants (crop scales, interpolation modes, compression levels), together with the image's nearest-neighbor density among other evaluation images. A set encoder embeds each profile, mean-pools across the benchmark, and outputs a two-part benchmark-level latent. One part is a pipeline offset that should explain shifts common to all images. The other part is an estimated exposed fraction. We train the audit on pools drawn from some controlled backbones and pipelines, and test it on held-out backbones, held-out pipelines, and on the publicly released DINO and MoCov3 checkpoints.
The intuition for why pooling profiles should help is specific. Under mismatched preprocessing, every image's familiarity drops. Exposed images, however, should recover familiarity sharply once a re-processing variant approximates the pretraining pipeline, while unexposed images recover only mildly. That difference in shape across the grid is a property of the set, not of any single score.
Baselines
The naive reference point is the evaluation everyone currently runs. It reports many-way accuracy on public self-supervised checkpoints with whatever preprocessing the evaluation code uses, and makes no contamination adjustment. On our controlled backbones, this corresponds to the fully exposed, matched-pipeline condition. Our claim is beaten if the no-exposure condition matches it within bootstrap error at 100 and 200 ways.
For the measurement study, the strongest classifier is IbM2 from Instance-based Max-margin for Practical Few-shot Recognition. That paper reports top-1 gains of 0.5 to 2.0 points over linear baselines on 200-way CUB and 1000-way ImageNet-1K. It is the strongest because it is designed for exactly this regime and stabilizes estimates in 3 to 5 episodes. We run it with the authors' perturbation and binary-search procedure, using the same features, shots and episode counts as nearest-neighbor and linear probing. For reference, nearest-neighbor on frozen features reached 34.2% 1-shot top-5 on the ImNet split, against 39.0% for the hierarchical prediction net, in Large-Scale Few-Shot Learning. The best 100-way 1-shot DF20 result reported is 9.75%. The first claim is supported if $\Delta(w,\kappa)$ is positive and increasing in $w$ and decreasing in $\kappa$ for all three classifiers, including IbM2. If the effect is specific to the crudest classifier, we do not count it as support.
For the audit, there is no reported result found on Lacuna for benchmark-level contamination detection in frozen vision encoders. The strongest competitor is therefore one we construct. It averages a per-image membership score, the embedding stability under the evaluation pipeline's own augmentations, and subtracts the same average on a clean reference set processed identically. Detectors like that of Rethinking Pretraining Data Detection for LLMs report AUC gains of 1.3 to 1.5 points on WikiMIA. Those numbers come from text, so they indicate the scale of typical gains and are not a comparable target.
The audit gets the same profiles, the same training pools and the same reference images as this baseline. The audit must beat it in benchmark-level AUC and in error on the estimated exposed fraction, specifically under mismatched preprocessing. Two ablations isolate the key ingredient. The first gives the averaged baseline the full re-processing profile, which removes set-level pooling while keeping the profile. The second gives the set model only the single-pipeline score, which keeps pooling and removes the profile.
Why not just…?
Why not just deduplicate the evaluation images against the pretraining corpus? When the corpus is available, exact and near-duplicate matching is the right first step, and we include it. It is a serious threat to the audit's usefulness for open corpora. It does not apply to closed or undocumented corpora, which is the common case for public checkpoints. It also cannot detect the class-level exposure condition, where the specific images are new but the classes were well covered. The measurement study does not depend on deduplication, because membership is known by construction.
Why not just average per-image membership scores, with a clean reference set? This is the most plausible competitor, and it may be sufficient whenever a same-distribution clean reference set exists and preprocessing matches. Our first ablation, which gives the averaged baseline the same re-processing profile, settles this directly. If the averaged baseline matches the set model, we report the simpler audit and drop the hierarchical latent. The measurement claims are unaffected.
Why not just collect fresh post-cutoff images, as in the chronological analysis of Task Contamination? Fresh data gives clean numbers going forward. It does not say how inflated existing reports are. It is also costly for fine-grained domains like fungi, where expert labels are scarce. Our controlled exposure conditions quantify the inflation that fresh data would only remove.
Why not just standardize preprocessing, as The Devil is in the Details and the evaluation-as-a-service model of Infrastructure support for evaluation as a service do? We standardize our own evaluation pipelines. The backbone's pretraining pipeline, however, is fixed by whoever trained it and is often undocumented. Standardizing evaluation therefore removes variance across papers, but not the mismatch between pretraining and evaluation that, by our second claim, hides contamination.
Decisive Checks
Three results would change our conclusions. If $\Delta(w,\kappa)$ is flat in the number of ways, the first claim fails. In that case contamination inflates many-way and low-way benchmarks alike, and existing ranking studies are less threatened than we suspect.
If mismatched preprocessing does not reduce the averaged membership score of exposed pools relative to matched preprocessing, the confound in the second claim does not exist. Simple audits then suffice.
If the audit's benchmark-level AUC under mismatch on held-out public checkpoints does not exceed the profiled averaged baseline, we abandon the hierarchical component. We then publish the controlled measurements and the simpler audit as the contribution.
References
- Few-Shot Image Classification Benchmarks are Too Far From Reality: Build Back Better with Semantic Task Sampling
- Instance-based Max-margin for Practical Few-shot Recognition
- Task Contamination: Language Models May Not Be Few-Shot Anymore
- Rethinking Pretraining Data Detection for LLMs: From Local to Global
- The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation
- Let's Stop Incorrect Comparisons in End-to-end Relation Extraction!
- Towards a Neural Statistician
- Unsupervised learning of global factors in deep generative models
- Large-Scale Few-Shot Learning: Knowledge Transfer With Class Hierarchy
- Infrastructure support for evaluation as a service