Choosing Feature Depth Per Task for Cross-Domain Few-Shot Prototypes

Image classifiers pretrained on natural photos learn general features such as edges and textures in their early layers and source-specific concepts in their late layers. When such a network is asked to recognize chest X-rays or skin lesions from five labeled examples per class, the usual recipe still averages the examples' final-layer features into one "prototype" per class and labels new images by their nearest prototype. That recipe fails badly on the most distant domains. This note proposes to let the few labeled examples themselves decide which depth of the network to trust. For each new task, we measure at every layer how well the main directions of variation in the support features line up with the support labels. We then weight per-layer prototype classifiers by that measure, with no extra training and no target-domain validation set.

The opportunity is concrete. A Broader Study of Cross-Domain Few-Shot Learning found that episodically meta-learned models, prototypical networks among them, do worse on ChestX than a network with random frozen weights. Separately, Representation Alignment in Neural Networks shows that how well a layer's dominant feature directions match the labels rises with depth on the training task, and that pretraining on a dissimilar task can push those directions away from what a new task needs. Together these results suggest that the best layer for a prototype classifier moves shallower as the target domain moves further from the source. They also suggest the shift can be detected from labeled support data alone. Nobody in the material has turned that diagnostic into a classifier.

Core Claim

Our claim is that a support-set estimate of representation alignment, computed separately at each layer of a frozen pretrained backbone, predicts which layer's nearest-prototype classifier will generalize on that episode. Weighting layers by this estimate should beat both the final-layer prototype and a fixed concatenation of all layers on the far-domain targets ChestX and ISIC. It should not hurt on the near-domain target CropDiseases. Two things would be new: depth is treated as a per-task quantity rather than a global design choice, and it is set by a training-free statistic that does not need held-out target labels.

Hold-out labels matter here. A Comparison of Machine Learning Methods for Cross-Domain Few-Shot Learning documents that tiny validation splits make tuning in this regime unreliable, often worse than default settings.

Mechanism

For a few-shot episode with $N$ classes and $n$ labeled support images, let $H_\ell \in \mathbb{R}^{n \times d_\ell}$ be the globally pooled, $\ell^2$-normalized, centered support features at layer $\ell$. Let $Y \in \mathbb{R}^{n \times N}$ be the centered one-hot label matrix.

Because $n \ll d_\ell$, we work with the $n \times n$ similarity matrix $H_\ell H_\ell^\top$. Its top $N$ eigenvectors $U_\ell$ span the directions of greatest support variance. The alignment score is the fraction of label energy captured by those directions:

$$a_\ell = \frac{\lVert U_\ell^\top Y \rVert_F^2}{\lVert Y \rVert_F^2}.$$

This score is deliberately not a separability test. In high dimensions almost any layer can perfectly separate 25 points, so separability carries little information. What the score asks is whether the layer's dominant variation is about the classes or about something else, such as scanner contrast or source-domain object identity.

Each layer gets a nearest-centroid classifier using squared Euclidean distance on normalized features, as in Prototypical Networks for Few-shot Learning. The combined score for query $x$ and class $k$ is

$$s_k(x) = -\sum_\ell w_\ell \, \lVert f_\ell(x) - c_{k,\ell} \rVert^2, \qquad w_\ell \propto \exp(a_\ell / \tau).$$

Here $f_\ell$ is the pooled layer-$\ell$ feature, $c_{k,\ell}$ is the class-$k$ support mean at that layer, and $\tau$ is a single temperature fixed once on source-domain (miniImageNet) episodes and never retuned on any target.

Relation to Prior Work

Two papers motivate the proposal. Representation Alignment in Neural Networks supplies the statistic, which it uses to analyze training dynamics; we instead use it as a per-episode layer selector at test time. Representation Similarity Analysis for Efficient Task Taxonomy & Transfer Learning shows that early layers are shared across visual tasks and that task relatedness lives at the encoder's end. It compares tasks to each other, while we compare layers within one small labeled task.

The closest method is Layer-Wise Feature Metric of Semantic-Pixel Matching for Few-Shot Learning. It pulls features from the last two ResNet layers and matches query pixels to support pixels with Hungarian assignment. Its layer choice is fixed empirically, it builds no class prototypes, and it was evaluated only on natural-image benchmarks. Our proposal spans all stages, uses class-level prototypes, and chooses depth adaptively under extreme domain shift.

Hierarchical prototype methods connect hierarchy and prototypes inside a single final embedding: HiPoSeg with whole-to-part prototypes, the Concept Subspace Network with parallel concept subspaces, HAPPI with hyperbolic generic-to-specific parts, and DeepTaxonNet with tree-structured latent clusters. All of them need a label tree or a large training set, and none is tested in few-shot cross-domain transfer. We do not use a label hierarchy. The hierarchy we exploit is the network's own general-to-specific gradient across depth. This choice is deliberate: BSCD-FSL targets offer no shared ontology, and that is precisely the regime where flat prototypes collapse.

TRGP provides layer-wise evidence that low layers transfer while high layers diverge, but it does so for continual learning with gradient projection and stored SVD bases. Our approach stores nothing and trains nothing.

Research Program

The program has three parts. First, we test whether the statistic is predictive before building anything on it. We compute $a_\ell$ at each residual stage of ImageNet- and miniImageNet-pretrained ResNet-10/18 backbones and at several ViT-Small blocks, on 600 episodes per BSCD-FSL target at 1, 5 and 20 shots. On each episode we also measure the query accuracy of every single-layer prototype classifier and compute the within-episode rank correlation between the two.

Second, if that correlation holds, we evaluate the weighted classifier against the baselines below. We also describe how the selected depth moves across CropDiseases, EuroSAT, ISIC and ChestX. As a secondary transfer test, we apply the same frozen-feature procedure to the 1-shot Hematopoiesis cell-stage benchmark from Few Shot Hematopoietic Cell Classification, with no retuning.

Third, we probe the failure mode. Alignment estimates from 5 or 25 points are noisy. We will measure how often the 1-shot estimate selects the wrong layer and whether shrinking weights toward uniform at low shot helps. That shrinkage, if used, will also be fixed on source episodes.

Baselines

The naive reference is a frozen-backbone nearest-centroid cosine classifier on the final pooled layer. It is the default in Meta-Baseline, and it is the thing we claim depth adaptation fixes. Next to it we run a randomly initialized frozen backbone with the same classifier. A Broader Study of Cross-Domain Few-Shot Learning reports that this random-feature reference beats meta-learning on ChestX, so any method that cannot clear it on ChestX has failed.

The strongest frozen-feature competitor is $\ell^2$-regularized one-vs-rest logistic regression on normalized final-layer features. A Comparison of Machine Learning Methods for Cross-Domain Few-Shot Learning found it and the cosine classifier to be the most accurate shallow methods on these targets. No per-dataset accuracy for it was found on Lacuna; the paper only says ChestX accuracy sometimes approaches chance. We run it with the default regularization strength, because that paper shows tuned settings are unreliable here. We also run a variant on concatenated all-layer features, so that classifier capacity and layer access are separated.

The strongest adapting competitor is whole-network fine-tuning on the support set. A Broader Study of Cross-Domain Few-Shot Learning reports that fine-tuning averages 12.8% higher accuracy than meta-learning across the four targets. No per-target fine-tuning figure is given in the file.

Among recent methods, REAP from Random Registers for Cross-Domain Few-Shot Learning reports an average 5-way 5-shot accuracy of 66.75% across CropDiseases, EuroSAT, ISIC2018 and ChestX with a ViT. That figure is the bar for our ViT-Small runs. For 1-shot CropDiseases, Adversarial Task Augmentation reports 77.82% with TPN.

Fairness rules apply to all comparisons. Every method uses the same backbone weights, the same 600 fixed episodes, and the same pooling. No method receives target-domain tuning beyond what its original paper used. Fine-tuning gets the step count and learning rate from the BSCD-FSL protocol, applied identically on every target.

The ablations isolate the key ingredient, which is support-set alignment weighting. Uniform layer weights test whether depth access alone does the work. A Fisher-ratio separability weight replaces alignment with a criterion that should overfit in high dimensions. A per-episode oracle picks the layer that is best on the query set and gives an upper bound. A per-dataset oracle picks the single layer that is best on held-out target episodes.

Beating the field means three things at once: clearing the logistic-regression and concatenation baselines on ChestX and ISIC by more than the 95% episode confidence interval, matching them on CropDiseases, and reaching REAP's 66.75% four-target average with a ViT-Small backbone and no training.

Why not just…?

Why not just concatenate features from all layers and fit logistic regression? This is the most direct threat, and the uniform-weight and concatenated-logistic baselines test it head-on. Concatenation lets the classifier find useful directions, but 5 to 25 points spread over thousands of mixed-depth dimensions give it little to estimate from. Concatenation also cannot down-weight a deep layer whose dominant variance is driven by the source domain. If concatenated logistic regression matches our method on ChestX and ISIC, the alignment weighting adds nothing and we will report that.

Why not just fine-tune the whole network on the support set? Fine-tuning is the strongest reported recipe in A Broader Study of Cross-Domain Few-Shot Learning. At the same time, Overcoming Data Scarcity in Specialized Medical Deep Learning and the chest X-ray analysis report that high-capacity models adapted on tiny cohorts latch onto acquisition artifacts. Our method changes no weights and costs one small eigendecomposition per layer. The two are also complementary: alignment scores can choose which layers to unfreeze. We will report fine-tuning both alone and restricted to the layers our score favors.

Why not just pick the best layer once per target dataset? If a practitioner knows the domain and has labeled held-out episodes, a fixed per-dataset layer may capture most of the gain. The per-dataset oracle ablation measures exactly this. If our method falls within noise of that oracle, its value is replacing a target validation set the regime does not have. If it beats the oracle, the per-episode adaptivity matters, for example across ISIC lesion types.

Why not just self-supervise on unlabeled target images? Understanding Cross-Domain Few-Shot Learning Based on Domain Similarity and Few-Shot Difficulty shows that target-domain self-supervision can win when the domain gap is large. It reports gains of up to 27.50 points on EuroSAT with strong augmentation, but it needs substantial unlabeled target data and loses on hard few-shot targets. Our setting assumes only the support set. Where unlabeled data exist, the two methods stack: self-supervision changes the backbone, and our selector chooses depth within it.

Decisive Checks

Three checks decide the idea. The first is a precondition. Within-episode rank correlation between $a_\ell$ and single-layer query accuracy must be clearly positive on ChestX and ISIC at 5 shots; if it is near zero, the proposal stops there. The second is mechanistic. The alignment-weighted mean depth should decrease monotonically from CropDiseases through EuroSAT and ISIC to ChestX, matching the domain-distance ordering in the BSCD-FSL design. That pattern would be direct evidence that the meta-learning collapse is partly a problem of using the wrong depth. The third is the bottom line. The weighted prototype classifier must beat penultimate-layer logistic regression and uniform concatenation on the far domains while matching them on the near domain, with $\tau$ never touched on a target.