A prototype classifier labels a new image by comparing its embedding to one average embedding ("prototype") per class, where the averages come from a handful of labeled examples. Prototype classifiers built on frozen pre-trained networks are now the default strong baseline for recognizing large numbers of new classes. They beat elaborate continual-learning methods (Revisiting Class-Incremental Learning with Pre-Trained Models) and episodic meta-learners once the label space reaches the thousands (Large-Scale Few-Shot Learning: Knowledge Transfer With Class Hierarchy). The catch is that the "new" classes are often not new to the backbone. Pre-training corpora quietly contain them. When overlapping classes were removed from video benchmarks, few-shot accuracy fell by up to 17.1% (A New Split for Evaluating True Zero-Shot Action Recognition). No one has asked how this hidden exposure interacts with the number of classes, which is exactly the variable that many-class ("high-way") evaluation is meant to stress.
We propose that pre-training exposure changes the shape of the accuracy-versus-number-of-classes curve, not just its height. Classes the backbone has memorized stay separable as more competitors are added, while genuinely new classes crowd together. If that holds, the decay profile can be measured from the labeled support set alone, by subsampling classes and computing leave-one-out prototype accuracy. That profile then gives a cheap dataset fingerprint of the kind AutoML uses to characterize tabular datasets. It can flag which classes are leaked, estimate how much the reported many-class accuracy is inflated, and tell a practitioner whether frozen prototypes will hold up or whether the backbone needs adaptation. It does all this without access to the pre-training data.
Core Claim
Our first claim is about scaling. For a frozen backbone $f$ and a set of $N$ classes with $k$ support examples each, let $A(n)$ be the leave-one-out nearest-prototype accuracy when only $n \le N$ randomly chosen classes compete. Leave-one-out accuracy means that each support example is classified against prototypes built without it. We claim that $A(n)$ decays measurably more slowly for classes the backbone was exposed to during pre-training than for classes it never saw, even when the two groups start at similar accuracy in the easy 5-class setting.
Our second claim is that two features computed on the support set carry this signal well enough to detect leakage without any pre-training metadata. The first is the slope of $A(n)$ against $\log n$. The second is a relative landmark, meaning the ratio of the target backbone's profile to that of a "control" backbone known not to have seen the classes.
The first claim is falsifiable on its own. If leaked and clean classes produce parallel decay curves, leakage is a constant offset. Many-class benchmarks would then not be specially distorted by it, and the fingerprint reduces to ordinary accuracy.
Where This Sits
The proposal borrows its method from dataset characterization. Landmarking, which means probing a dataset with cheap learners and using their performance as features, was shown to find better-matched historical tasks than simple statistics (Using Meta-Learning to Initialize Bayesian Optimization of Hyperparameters). Relative landmarking, which compares a cheap learner against a reference, predicted when tuning a support vector machine beats its defaults with AUC around 0.80 (A meta-learning recommender system for hyperparameter tuning). Standard meta-feature taxonomies such as MFE target tabular data (Characterizing classification datasets). The closest learned selector, OOD-Chameleon, characterizes datasets by size and distribution-shift measures to pick a robust training algorithm. None of these characterizes an embedding-space task, and none targets pre-training exposure. Our probe is a nearest-prototype landmarker run at several class counts, and the control backbone plays the reference role that the default model plays in relative landmarking.
Two lines of leakage work motivate the proposal. TruZe identified overlapping classes with a combination of text-embedding similarity and visual prediction checks, and confirmed attribution with a handcrafted-feature control that did not change across splits (A New Split for Evaluating True Zero-Shot Action Recognition). Revisiting Few-Shot Object Detection with Vision-Language Models showed zero-shot GroundingDINO at 48.3 AP on COCO novel classes against 33.1 AP for a dedicated few-shot detector. Both works detect leakage after the fact and at the level of label names. They say nothing about degree of exposure, and they cannot work when the pre-training label set is unknown. We keep TruZe's control-feature logic and move it from post-hoc attribution to a support-set measurement.
The many-class literature constrains the design. A Closer Look at Few-Shot Classification with Many Novel Classes built ImageNet-MNC, with 15,000 ImageNet-21K test classes strictly disjoint from ImageNet-1K base classes. On it, the SHA-Pipeline (hubness normalization plus hierarchy-aware losses) lifts 1-shot 2,560-way accuracy from 12.70% for a plain prototype network to 18.91%. That strict disjointness is what makes a controlled leakage study possible here. Few-Shot Image Classification Benchmarks are Too Far From Reality showed that uniformly sampled tasks are semantically coarse (median coarsity 85.1 on tieredImageNet) and that fine-grained 100-way Danish Fungi tasks drop the best 1-shot model to 9.75%. This matters because a decay profile could reflect semantic granularity rather than exposure, so our analysis must control for coarsity.
Several works compete as classifiers but not as characterizers. Confusable Learning mines hard negatives from a running confusion matrix. IbM2 replaces centroids with instance-level max-margin classifiers for 1,000-way tasks. Meta-Baseline shows that cosine prototypes suit classification-pre-trained features. We do not propose a better classifier. We treat these methods as candidates the fingerprint must choose among.
Research Program
The program has three stages.
Stage 1: build ground truth for exposure. Starting from ViT-B backbones pre-trained only on ImageNet-1K, which by construction have never seen ImageNet-MNC test classes, we continue pre-training on a mixture that includes a random subset $S$ of MNC classes. The continued training uses images disjoint from the evaluation images, at controlled doses (roughly 10, 100, and all available images per class). We run it in two forms. The first is supervised with labels. The second is self-supervised without labels, which mimics unlabeled web exposure, the realistic case. This yields backbones with known, per-class, graded exposure. Every downstream task then mixes leaked classes from $S$ with clean classes.
Stage 2: measure and model the decay profile. For each task we compute $A(n)$ at $n \in \{5, 20, 80, 320, 1280, 2560\}$ by class subsampling on the support set, averaging over repeated draws. We then fit a slope $\beta$ by regressing $A(n)$ on $\log n$; the log scale is used because crowding-driven confusion in prior many-class results grows roughly with the number of competitors per neighborhood. Alongside the profile we record a per-class version: the rank of each class's own prototype among its nearest neighbors as $n$ grows. We also compute three standard geometry statistics:
- hubness, measured as the skew of the distribution of how often each prototype is someone's nearest neighbor;
- intra-class spread relative to the nearest-prototype distance;
- WordNet coarsity where a taxonomy exists.
The relative landmark repeats the profile under a control backbone that is guaranteed unexposed, here the Stage 1 starting checkpoint. It is normalized by the gap the two backbones show on a small set of classes known to be clean.
Stage 3: use the fingerprint. A small meta-model (gradient-boosted trees, to stay close to the Random Forest recommenders it descends from) maps the fingerprint to three outputs: per-class leakage probability, the predicted inflation of many-class accuracy relative to an exposure-free backbone, and the classifier choice.
Evaluation uses leave-one-backbone-out and leave-one-dataset-out splits, so the meta-model is never scored on the backbone or dataset it was trained on. The method needs $k \ge 2$ shots for leave-one-out accuracy. The 1-shot regime is therefore handled only through augmentation-based views and is reported as a stated limitation, not a headline.
Baselines
The naive reference point is to do nothing: report the frozen prototype classifier's many-class accuracy at face value and assume no leakage. For the inflation question, this corresponds to predicting zero inflation. For selection, it corresponds to always using frozen cosine prototypes, the SimpleCIL recipe (Revisiting Class-Incremental Learning with Pre-Trained Models). That paper reports SimpleCIL beating prior methods by 5% on VTAB with no downstream training, which makes it a genuinely strong default. To beat it, the fingerprint-guided choice must achieve lower average regret than the oracle per-task choice, with the difference significant across held-out tasks.
For leakage detection, the strongest existing method is name-based overlap checking in the style of TruZe (A New Split for Evaluating True Zero-Shot Action Recognition): text-embedding similarity between target class names and the pre-training label vocabulary, combined with the backbone's own classification head where one exists. It is the strongest competitor because it is what the field actually uses. In our supervised-exposure condition it has the full label vocabulary, making it close to an oracle. We run it with the same class names and the same threshold-tuning budget on the same validation classes. On the TruZe video splits, where it defines the ground truth, it is used only for scoring. No reported AUROC for name-based leakage detection was found on Lacuna. Beating it therefore means matching it within a few AUROC points in the supervised-exposure condition and clearly exceeding it in the unlabeled self-supervised-exposure condition, where name matching has no signal.
For predicting at-scale accuracy and choosing classifiers, the strongest methods are the classifiers themselves, each run on identical frozen features, support sets, and hyperparameter budgets. The candidates are the SHA-Pipeline (A Closer Look at Few-Shot Classification with Many Novel Classes), with a reported 18.91% at 1-shot 2,560-way on ImageNet-MNC against 12.70% for a plain prototype network; IbM2 (Instance-based Max-margin for Practical Few-shot Recognition), reported at 0.5% to 2.0% over linear classifiers on 1,000-way tasks; a linear probe; and APER-style adapter tuning. We found no reported 5-shot 2,560-way ImageNet-MNC number on Lacuna, so our own reruns define that reference.
Two ablations isolate the key ingredient. One fingerprint drops the decay slope and keeps only static geometry (hubness, spread, coarsity). The other drops the control-backbone ratio. If the static-only version performs as well, the scaling claim adds nothing.
Why not just…?
Why not just match class names against the pre-training label list? Where the list exists, this is a serious threat, and we expect it to win on binary class-level overlap. It fails in three places this proposal targets. Name matching cannot grade exposure; TruZe's own authors could only remove classes, not quantify partial overlap. It cannot see unlabeled or mislabeled exposure, which is the norm in web corpora. And it is undefined for backbones with undisclosed data. The self-supervised-exposure condition in Stage 1 tests exactly this, because there the names carry no signal while the images still leak.
Why not just use zero-shot accuracy from a vision-language model as the leakage score? High zero-shot accuracy is equally consistent with memorization and with genuine semantic transfer. Indeed, Revisiting Few-Shot Object Detection with Vision-Language Models argues for concept alignment precisely because the two are entangled. The method also does not apply to image-only backbones. We include zero-shot CLIP accuracy as a feature and as a standalone baseline on the vision-language backbones. If it matches the decay-slope fingerprint there, we will report that the cheaper signal suffices for that model family.
Why not just evaluate every candidate classifier by cross-validation on the support set and pick the winner? For selection with $k \ge 5$, this is a real competitor, since it uses the same support data. It has two costs. Running IbM2, SHA-style training, and adapter tuning under cross-validation at 2,560 ways multiplies compute severalfold, while the fingerprint needs only a pass of prototype computations. More importantly, cross-validation says which classifier fits the support set best, not whether that accuracy comes from memory. It therefore cannot answer the inflation question at all. We run support-set cross-validation as a selection baseline under an equal wall-clock budget. If it wins on regret, we will narrow the selection claim and keep the leakage claim, which cross-validation does not address.
Why not just use hubness normalization and move on? Z-score hubness correction improves accuracy by roughly 0.9% on ImageNet-MNC (A Closer Look at Few-Shot Classification with Many Novel Classes). But it repairs geometry without diagnosing why that geometry arose. We apply it to all classifiers, and we include hubness in the static ablation so its contribution is measured rather than assumed.
Proposed Experiments
The decisive first check is the scaling hypothesis on controlled exposure. Using the Stage 1 backbones, we plot $A(n)$ for leaked and clean MNC classes. The two groups are matched on 5-way accuracy and on WordNet coarsity, so that granularity cannot explain the result. If the slopes do not differ at any dose, the scaling claim is refuted. We would then fall back to reporting leakage as a level effect, and drop the decay slope from the fingerprint. A handcrafted or randomly initialized feature control should show no leaked-versus-clean difference, mirroring TruZe's IDT control.
The second experiment tests detection transfer to naturally leaked data. ImageNet-21K-pre-trained public backbones are exposed to MNC classes by construction, while 1K-pre-trained ones are not. The TruZe UCF101 and HMDB51 splits supply real overlap labels against Kinetics-400. We measure AUROC for per-class leakage against the name-matching baseline, and rank correlation between predicted and measured inflation. Measured inflation is the accuracy gap between the exposed backbone and the control on the same tasks.
The third experiment tests selection across regimes. The task suite spans ImageNet-MNC at 80 to 2,560 ways, fine-grained Danish Fungi tasks drawn with semantic task sampling, and the uncontaminated class-incremental suite (ImageNet-A, ObjectNet, OmniBenchmark, VTAB). On this suite we compare fingerprint-guided choice against always-frozen prototypes, global-best, and support-set cross-validation, measuring accuracy regret relative to the per-task oracle.
The result that would most change practice is if leaked and clean benchmarks rank these classifiers differently at high way counts. For example, frozen prototypes might look sufficient on leaked tasks while adaptation or max-margin classifiers win once exposure is removed. That would mean current many-class leaderboards partly measure pre-training coverage rather than scalability.
References
- Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need
- Large-Scale Few-Shot Learning: Knowledge Transfer With Class Hierarchy
- A New Split for Evaluating True Zero-Shot Action Recognition
- Using Meta-Learning to Initialize Bayesian Optimization of Hyperparameters
- A meta-learning recommender system for hyperparameter tuning: Predicting when tuning improves SVM classifiers
- Characterizing classification datasets: a study of meta-features for meta-learning
- OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable?
- Revisiting Few-Shot Object Detection with Vision-Language Models
- A Closer Look at Few-Shot Classification with Many Novel Classes
- Few-Shot Image Classification Benchmarks are Too Far From Reality: Build Back Better with Semantic Task Sampling
- Confusable Learning for Large-Class Few-Shot Classification
- Instance-based Max-margin for Practical Few-shot Recognition
- Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning