Active learning lets a model pick which unlabeled examples a human should label next. It breaks down at the very start. Before any labels exist, the model has no basis for judging which examples are informative. The standard fix is to skip the model and pick examples that cover the shape of the data in some pre-existing feature space, such as text-model embeddings, CLIP image embeddings, or propagated graph features. This only works if that feature space organizes the data by the things the task cares about. A separate line of work on parameter-efficient adaptation (training small add-on modules while the large pretrained network stays fixed, or "frozen") keeps finding the opposite in specialized domains. Frozen features from natural-image or even domain-specific pretraining often lack the directions needed to separate target classes, and they often encode scanner, stain or compression artifacts more strongly than biology. We call this representation rigidity. A cold-start selector that maximizes coverage in such a space will spend its label budget covering artifacts and will under-sample exactly the regions the frozen network cannot resolve.
We propose a label-free correction to the space in which the first labels are chosen. It has two parts. First, we estimate and project out the directions in the frozen embedding that move under nuisance perturbations such as recompression, stain and color shifts, and blur. Second, we add a small pool-specific residual feature, trained by self-supervision on the unlabeled pool without backpropagating through the backbone. This feature restores resolution where the frozen map collapses distinct images together. The same two quantities give a label-free rigidity score. The claim we will test is that this score predicts, before any labeling, when coverage in the raw frozen space will do worse than random sampling, and that selecting in the corrected space closes that gap for the few-shot adapters practitioners actually use.
Core Claim and What Is New
Cold-start methods all select in an assumed-trustworthy embedding. PATRON propagates zero-shot prompt uncertainty over sentence embeddings and partitions them with K-Means. DEUCE builds a graph that combines semantic neighbors with zero-shot-prediction neighbors, then samples representative-and-uncertain points. STENCIL runs submodular mutual-information selection over word-vector embeddings. FeatProp clusters graph-smoothed raw node features. None of them asks whether the embedding is fit to select in. These papers motivate the proposal by showing that coverage-based cold start works when the space is good.
The rigidity literature supplies the reason to doubt that assumption in specialized imaging. Few-shot Adaptation of Medical Vision-Language Models finds that adaptation gains are capped by the frozen backbone's prior grasp of the modality. Path-CLIP shows that CLIP features lack sensitivity to cellular texture. Most pointedly, the NCT-CRC-HE audit shows that a widely used histology benchmark can be classified at 82.2% from color histograms alone, which means acquisition artifacts are strongly structured in the data.
The closest existing work is the chest radiograph cold-start study. It compares domain-specific and ImageNet backbones as the basis for cold-start initialization. It found that domain-specific models did not beat ImageNet in 6 of 8 initialization experiments, and that frozen embeddings could not substitute for pixels in 7 of 8. That study diagnoses the problem by choosing among backbones. It does not change the selection geometry, does not predict failure without labels, and does not pair selection with modern adapters. Our proposal does all three.
Research Program
Let $f(x)$ be the frozen backbone embedding of image $x$, and let $\mathcal{T}$ be a family of label-preserving nuisance transforms. In histology these are JPEG recompression at varied quality, stain and color perturbation, and mild blur and rescaling. In radiographs they are contrast windowing, noise, and resolution changes. We form the covariance $C$ of the displacements $f(t(x)) - f(x)$ over pool images and sampled $t \in \mathcal{T}$. We take its top-$k$ eigenvectors $N$ as the nuisance subspace and define the selection embedding
$$z(x) = \big[(I - NN^\top) f(x)\,;\ \lambda\, g_\phi(x)\big],$$
Here $g_\phi$ is a small residual branch. In the spirit of the detached bypass in Res-Tuning, it reads intermediate backbone activations and is trained with no gradient into the backbone. It is trained on the unlabeled pool with a self-supervised objective that is invariant to $\mathcal{T}$. The weight $\lambda$ is set by matching variances, not by labels.
Selection then runs a standard coverage procedure in $z$. We will use K-Medoids, as in FeatProp, so that the only change from the strongest simple baseline is the space being selected in.
The rigidity score combines two label-free quantities. The first is the fraction of the variance of $f$ that lies in $N$, which measures how much the space is about nuisance. The second is the disagreement between the k-nearest-neighbor graphs built from $f$ and from $g_\phi$, which measures how much pool-specific structure the frozen map fails to resolve.
We will build three things. The first is the selection pipeline. The second is a study across backbone and dataset pairs that relates the rigidity score to the cold-start gap. The third is a controlled artifact-injection testbed in which a label-independent nuisance is planted at known strength.
Baselines
The naive reference is uniform random sampling of the same budget. It is run with the same downstream learner and averaged over many seeds, because PATRON and LP+text both report high seed variance at few shots. Beating random on average is necessary but not sufficient; we also require lower variance across seeds.
The simplest serious competitor is K-Medoids or K-Center coverage in the raw frozen embedding $f$. This is what a practitioner would try first, and it isolates our contribution exactly, since the procedure is identical and only the space differs. We will also run the diversity and hybrid initializations from the chest radiograph study on its Guangzhou and Pakistan datasets. That study reports only win counts, not accuracies, so there is no reported number to reproduce and we will regenerate its baselines ourselves.
The strongest existing method is a vision port of DEUCE. The medical vision-language backbones (Quilt-1M, FLAIR, MedCLIP) have zero-shot heads, so DEUCE's dual-neighbor graph can be built from image-embedding neighbors plus zero-shot-prediction neighbors, followed by its density-based propagation and farthest-point sampling. DEUCE reports a 6.2% gain on imbalanced TREC and 2.5% on balanced text sets over random, entropy and coreset baselines. No reported result was found on Lacuna for DEUCE or PATRON on any image benchmark, so we will establish this baseline ourselves with the authors' hyperparameters and an equal tuning budget.
We also include PT4AL-style rotation-loss ranking (PT4AL), which reported 55.83% versus 46.02% for random at iteration 1 on CIFAR10. It is the only cold-start baseline in the file that already uses pool self-supervision.
Every selector feeds the same downstream learners: a frozen linear probe, LP+text, and a residual adapter. The budget is $S \in \{1,2,4,8,16\}$ labels per class, with identical training recipes.
The reference number for the downstream learner comes from LP+text under random 1-shot selection on NCT-CRC: 55.6% accuracy, against 52.05% for the linear probe and 46.05% for CoOp (Few-shot Adaptation of Medical Vision-Language Models). Beating it means exceeding the DEUCE port, with paired seeds, at 1- and 2-shot on at least the high-rigidity pairs.
The ablations do three things. They drop the projection, drop the residual feature, and replace $N$ with random directions of equal rank, so that any gain cannot be explained by dimensionality reduction alone.
Why not just…?
Why not just label a random seed and start active learning once the adapter is warm? At one to four labels per class there is no "warm" phase to wait for. The budget is the cold start. STENCIL reports 17–40% rare-class F1 gains over random at cold start under 1:10 to 1:18 imbalance, because random seeds miss minority classes. The random baseline will show directly whether this holds for imaging. If random matches our method on low-rigidity pairs, that is consistent with our claim, not a refutation of it.
Why not just cluster the frozen embeddings, or use a domain-specific backbone? These are the real alternatives, and the experiments are built to compare against them. The chest radiograph study suggests that a domain-specific backbone does not reliably fix initialization. The NCT-CRC audit shows that artifact structure exists to be clustered. If the rigidity score does not predict where raw-space coverage loses to random, or if raw-space coverage ties our method on Quilt-1M histology, the claim fails.
Why not just normalize stains and harmonize JPEG quality before embedding? This is the most serious threat. For known artifacts, preprocessing might remove the same directions our projection removes. We will run stain normalization plus recompression to a fixed quality as its own baseline. If it matches our method, the honest conclusion is that the nuisance-projection half adds nothing beyond preprocessing. The residual-resolution half would then have to carry the contribution, and the ablations separate the two. The planted-artifact testbed also includes a nuisance that no standard preprocessing targets, which tests whether learning the subspace from transforms generalizes beyond hand-designed fixes.
Why not just self-supervise the whole backbone on the pool and cluster that? This is a fair upper reference, and we will run it at matched wall-clock time. Its cost is the kind that Res-Tuning reports cutting by 49.7–70% in memory, and Path-CLIP warns that full updates on small domain data can degrade pretrained features. If it wins clearly, our contribution narrows to the cheap rigidity diagnostic.
Proposed Experiments
We will study backbone and dataset pairs chosen to span a range of rigidity. For histology: CLIP versus Quilt-1M on NCT-CRC-HE-100K, evaluated on the independent CRC-VAL-HE-7K patients, plus SICAPv2, SkinCancer, PCam and MHIST. For ophthalmology: FLAIR on MESSIDOR. For radiology: ImageNet DenseNet versus TXRV and REMEDIS on the Guangzhou and Pakistan radiographs. As a low-rigidity control, we use an ImageNet backbone on CIFAR10, where we predict our method roughly ties raw-space coverage. The metrics are balanced accuracy and macro-F1 by shot count, seed variance, the number of classes covered by the seed set, and, on the testbed, the fraction of budget spent on planted-artifact clusters.
The first decisive check is predictive. Across all pairs, the label-free rigidity score should correlate positively with the gap by which random beats raw-space coverage, measured with rank correlation over pairs. The second decisive check is causal. On the planted-artifact testbed, raw-space coverage and the DEUCE port should increasingly spend budget on artifact clusters as artifact strength grows, while our selection should not, and downstream LP+text accuracy should track that difference.
If neither check holds, the premise that rigidity corrupts cold-start selection is wrong for these domains. That would itself be a useful negative result for anyone using frozen foundation embeddings to choose their first labels.
References
- Cold-Start Data Selection for Better Few-shot Language Model Fine-tuning: A Prompt-based Uncertainty Propagation Approach
- Deuce: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learning
- STENCIL: Submodular Mutual Information Based Weak Supervision for Cold-Start Active Learning
- Active Learning for Graph Neural Networks via Node Feature Propagation.
- Few-Shot Adaptation of Medical Vision-Language Models
- Bridging the Pathology Domain Gap: Efficiently Adapting CLIP for Pathology Image Analysis with Limited Labeled Data
- NCT-CRC-HE: Not All Histopathological Datasets are Equally Useful
- Rethinking Domain-Specific Pretraining by Supervised or Self-Supervised Learning for Chest Radiograph Classification: A Comparative Study Against ImageNet Counterparts in Cold-Start Active Learning
- Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone
- PT4AL: Using Self-supervised Pretext Tasks for Active Learning