Choosing Which Few Target Images to Label When the Frozen Backbone Cannot See the Target Domain

When a model pretrained on everyday photos is adapted to X-rays, satellite tiles or skin-lesion images with only a handful of labels, two decisions matter. The first is how to adapt it. The second, which gets far less attention, is which handful of images to pay an expert to label. Active learning is the study of choosing which examples to label. Its diversity-aware methods pick images that are spread out in the model's feature space, meaning the internal vector the network computes for each image. That strategy quietly assumes distances in that space mean something. Under large domain shift the assumption breaks. A frozen pretrained network maps unfamiliar images onto a crowded, poorly separated region of its final feature space, a failure called representation rigidity, and "spread-out" points chosen there are spread along noise.

We propose to separate the space used to choose labels from the space used to classify. Images to label are selected in a cheap, label-free embedding built from intermediate layers of the frozen network, re-centred using statistics of the target images themselves. The classifier is adapted wherever it works best. We also propose a label-free rigidity score that decides, per target domain, how much to trust the final layer for selection. The claim is testable: the advantage of selecting in the re-centred intermediate space should grow with domain distance, from near zero on crop-disease photos to largest on dermoscopy and radiology. Selecting diverse points in the final frozen space should help less, or even hurt, as the shift grows.

Core Claim

Cross-domain few-shot learning has so far treated the labelled support set as given at random. A Broader Study of Cross-Domain Few-Shot Learning builds the benchmark of CropDiseases, EuroSAT, ISIC2018 and ChestX. It shows that plain fine-tuning beats episodic meta-learning by 12.8% on average and that meta-learners fall below a random frozen network on ChestX. Every method in the research file that uses this benchmark samples support images uniformly: normalisation fixes, distillation, and token-alignment methods for CLIP (the contrastively pretrained image–text model).

Meanwhile, diversity-aware active learning measures diversity in the final feature space. TypiClust-Margin in Bridging Diversity and Uncertainty in Active learning with Self-Supervised Pre-Training clusters final features. So does the density-aware core-set of In Defense of Core-set. The furthest-from-centroid rule of O-MedAL does the same. The TypiClust-Margin authors explicitly leave open how their schedule behaves when the backbone lacks coverage of the target domain.

Our claim is that under large shift the final frozen features are the wrong place to measure diversity, and that a better selection space already exists inside the frozen network. Two findings motivate this. Revisiting Learnable Affines for Batch Norm shows that source-learned batch-norm shifts push target activations below zero, where the ReLU nonlinearity (which zeroes negative values) silences them. They call this the "sparsity trap," and it means deep target features are partly erased before any classifier sees them. Cross-Level Distillation and Feature Denoising (CLDFD) finds that shallow layers carry transferable target information that deep layers lose. For a frozen network $f$ with intermediate layers $l = 1..L$, we define the selection embedding of an image $x$ as

$$z(x) = \big[\, \mathrm{pool}\big((h_l(x) - \mu_l)/\sigma_l\big) \,\big]_{l \in S}$$

Here $h_l(x)$ is the pre-activation output of layer $l$ (taken before the ReLU, so the trap cannot zero it). $\mu_l$ and $\sigma_l$ are per-channel mean and standard deviation over the unlabelled target pool. "pool" is spatial average pooling, $S$ is a chosen subset of layers, and the brackets denote concatenation. Diversity-aware selection is run on $z$. The classifier is then adapted with whatever method is strongest on the chosen labels.

Research Program

The setting is pool-based active few-shot learning. For a target dataset we hold an unlabelled pool of images and a total label budget $B$, drawn from $\{N, 2N, 5N, 16N\}$ for an $N$-way task. Unlike the standard protocol, class balance is not guaranteed. The labeller labels whatever images are chosen, so failing to cover a class is a real cost. This mirrors how a clinician or remote-sensing analyst would actually be asked to label.

The work has three parts. First, a measurement study. We run a fixed selection rule (TypiClust-style typical-point clustering, plus a core-set variant) in five spaces:

  1. final frozen features;
  2. final features re-centred on target statistics;
  3. our multi-layer pre-activation space $z$;
  4. $z$ with CLDFD-style top-$h$ activation pruning;
  5. the space of the model after one round of adaptation.

The fourth space tests a tension in the literature. CLDFD and ATHA gain by discarding weak activations or tokens, while density-aware active learning argues that informative points live in sparse regions. Pruning may therefore remove exactly the directions that separate unfamiliar target classes.

Second, a label-free rigidity score for each target domain and layer. It combines the fraction of channels silenced after ReLU across the target pool (the sparsity-trap signature) with the effective rank of the features (how many directions they really spread along). The score chooses the layer set $S$ and decides whether any final-layer signal enters selection. The prediction is that the score ranks the four benchmark domains in the known order of distance from natural images: CropDiseases, then EuroSAT, then ISIC, then ChestX.

Third, we cross selection with adaptation to see whether better selection survives better adaptation. The adapters are:

  • a linear probe;
  • Fine-Affine, which tunes only the normalisation scale and shift (Yazdanpanah et al.);
  • LoRA, which adds small low-rank weight updates;
  • LP+text (Shakeri et al.) and ATHA for CLIP.

Two backbones are used. The first is the ImageNet-pretrained ResNet of the benchmark, run through the open MLRC-2021 few-shot codebase. The second is CLIP ViT-B/16, the setting ATHA uses.

Uncertainty-based second rounds, in the style of TypiClust-Margin, are included only at the largest budget. At $B \le 5N$ there is essentially one acquisition round, so diversity is the whole question.

Baselines

The naive reference is random selection under the same budget, reported in two forms. Unbalanced random is what a practitioner without a method would do. Class-balanced random is the standard benchmark protocol, and it quietly uses label information our methods do not have, so it serves as a ceiling-like reference. Beating class-balanced random without seeing labels is the headline bar.

The strongest existing selection competitor is TypiClust-Margin run exactly as published on final frozen features. It is the method in the file most directly designed for small budgets on pretrained backbones, and it outperforms BADGE, core-set and ProbCover in its own evaluation, though the file gives curves rather than numbers ("no reported result found on Lacuna for" its per-dataset accuracy). A second selection competitor is an adaptation of MHPL from Active Source Free Domain Adaptation. MHPL picks uncertain points while forbidding neighbouring picks, in a single batch from the initial model. It is the closest prior work combining active selection with domain shift, reported at 79.1% on Office-Home, but it was designed for 5% budgets and full-network updates. We shrink its budget to ours and run it on the same frozen backbone.

The strongest adaptation competitors, both run on random supports, are CLDFD and ATHA. CLDFD reports 82.52% 1-shot accuracy on EuroSAT, against a previous best near 73%. It uses 20% of the unlabelled target data, the same kind of pool access we assume, which makes it a fair comparison. ATHA reports average gains over LoRA of +2.43% at 1-shot and +2.54% at 5-shot across the four target domains. The file gives no absolute ATHA accuracies, so we reproduce them.

Fairness is enforced by construction. Every method sees the same unlabelled pool and the same budget. Every method is evaluated with the same adapter and the same held-out test split, over at least 50 pool subsamples. Hyperparameters are tuned on CropDiseases only and frozen for the other domains, since tuning per domain would leak labels that a real active learner does not have.

The key ablation holds the selection rule fixed and changes only the space it runs in. If $z$ beats final-feature TypiClust under identical clustering, the gain belongs to the selection space and not to the clustering rule. Beating the strongest method means an accuracy gain at matched budget, on ISIC and ChestX, that exceeds the seed-to-seed standard error. It must also persist when the adapter is ATHA or LP+text, not only a linear probe.

Why not just…?

Why not just sample randomly, since few-shot budgets are tiny? At $B = N$ with no class-balance guarantee, random sampling frequently misses classes outright. Shakeri et al. note large variance across random few-shot draws. The evidence for active gains in the file, including O-MedAL's 92.02% on Messidor and DACS's 92.1% against 88.6% for core-set on RMNIST, comes from larger budgets and in-domain data, so it does not settle this regime. Our unbalanced and balanced random baselines at every budget answer it directly. If both random variants match our method on ISIC and ChestX, we report that selection is not worth doing there.

Why not just run TypiClust or core-set on the final features, as everyone does? This is the obvious idea, and our central hypothesis is that it degrades with domain distance. The sparsity-trap result says final target features are partly zeroed, and the file's summary of TypiClust-Margin says its schedule was never tested where the backbone lacks coverage. The space-only ablation settles this. The idea fails if final-feature TypiClust matches $z$ on ChestX.

Why not just fix the features first, with Feature Normalization, Fine-Affine or LP+text, and then sample randomly? This is the real threat. If better adaptation makes the choice of support images irrelevant, the proposal reduces to "use a good adapter." The full grid of selection method crossed with adapter tests it directly. We expect the selection gain to shrink but not vanish under stronger adapters, because no adapter can learn from a class it was never shown. If the gain vanishes under ATHA, we will report that.

Why not just train a self-supervised model on the target pool and select in its space? TypiClust relies on exactly this, and it may well beat us. It costs a full self-supervised pretraining run per domain, whereas $z$ costs one forward pass. We include SimCLR trained on each target pool as a compute-heavy reference, and we report accuracy together with compute. The honest outcome may be that $z$ recovers most of the self-supervised gain at a small fraction of the cost, not that it wins.

Proposed Experiments and Decisive Checks

All experiments use the four benchmark targets as 5-way tasks drawn from each dataset's classes. The ResNet backbone is pretrained on ImageNet or miniImageNet, following CLDFD. The CLIP backbone is ViT-B/16. As a second regime we use the medical vision-language benchmark of Shakeri et al.: NCT-CRC with the Quilt-1M backbone, where LP+text reports 55.6% 1-shot against 52.05% for linear probing, and CheXpert with MedCLIP. Metrics are:

  • test accuracy against label budget;
  • the number of classes covered by the selected set;
  • the variance of accuracy across pool subsamples.

We also report the Spearman correlation, across domains and layers, between the rigidity score and the advantage of $z$ over final-feature selection.

Three results would be decisive. First, the advantage of $z$ over final-feature TypiClust must increase monotonically from CropDiseases to ChestX. If it is flat, rigidity is not what limits selection. Second, the advantage must persist under ATHA and LP+text. If it does not, better adapters subsume selection. Third, the rigidity score must pick the best layer set on held-out domains without labels. If it does not, the method needs per-domain tuning, which defeats its purpose.

A fourth check addresses the pruning tension. If top-$h$ pruning of $z$ lowers class coverage on ISIC and ChestX while helping classification, that is direct evidence that denoising for classification and diversity for selection want different spaces. That would be a useful finding even if the headline gains are modest.

We expect ChestX to stay close to chance for every combination, as it does across much of the literature. A null result there, paired with a gain on ISIC, would still bound where active selection can and cannot compensate for a backbone that does not see the domain.