When a hospital adapts a large pretrained image model to a new diagnostic task using a few dozen labeled patients, it has to decide how much of the model to change. It can change nothing and only fit a linear classifier on frozen features. It can insert small trainable low-rank updates into the network's layers. Or it can retrain everything. The literature gives contradictory advice. Linear probing beats low-rank fine-tuning at 2 to 32 examples per class on chest X-ray prognosis, but loses once the full cohort is available. On the other hand, larger calibrated models are reported to rank better even on tiny subsets. Practitioners usually settle the choice with a validation split, and in cohorts this small that split can consume a third to half of the data.
We propose to let the training set itself say how many directions of change it can support. Before any fine-tuning, we compute the average gradient that the few labeled examples exert on each candidate layer. We then count how many of its singular values rise above the level that pure sampling noise would produce. That noise level is set by the Marchenko–Pastur law from random matrix theory, which predicts how large singular values get for a random matrix of a given shape. The count becomes the adapter rank for that layer. When the count is zero everywhere, the method falls back exactly to a text-anchored linear probe. The rule needs no validation data, contains the strongest few-shot baseline as a special case, and makes a falsifiable prediction about where the crossover between probing and fine-tuning occurs.
Core Claim
Let $W_l$ be a frozen weight matrix of layer $l$, of size $d_{out}\times d_{in}$. Let $G_l = \frac{1}{N}\sum_{i=1}^{N} g_i$ be the mean over the $N$ support examples of the per-example loss gradients $g_i$ with respect to $W_l$, evaluated at the text-anchored probe solution. Under the null hypothesis that the examples carry no consistent signal for this layer, $G_l$ behaves like a random matrix with entry variance $\sigma_l^2/N$. Here $\sigma_l^2$ is the per-example gradient variance, which we estimate from the same $N$ gradients. Its singular values then concentrate below the edge
$$ \lambda_+^{(l)} = \frac{\sigma_l}{\sqrt{N}}\left(\sqrt{d_{out}}+\sqrt{d_{in}}\right), $$
which is the standard Marchenko–Pastur bulk edge for a rectangular Gaussian matrix. We set the adapter rank to $r_l^* = \#\{k : s_k(G_l) > \lambda_+^{(l)}\}$, where $s_k(G_l)$ are the singular values of $G_l$. These are the directions in which the support set pushes the layer coherently, beyond what chance alignment of $N$ noisy gradients would produce.
The claim is that this count tracks the empirical crossover. It should be zero at very low shot counts on tasks where linear probing wins. It should grow with $N$, and grow faster on tasks where the frozen features are misaligned with the target. The adapters it sizes should match or beat the best fixed-rank adapter selected with hindsight, without spending any labeled data on validation.
Relation to Prior Work
Three empirical lines motivate the gap.
- Fine-tuning versus probing. Ruffini et al. compare full fine-tuning, LoRA, BitFit and linear probing across CNNs and foundation models on chest X-ray prognosis. They find probing more stable at 2–32 shots and low-rank fine-tuning competitive only on the full Stony Brook cohort. They use fixed ranks and offer no rule for locating the switch.
- Capacity versus shortcuts. Tartaglione et al. show that ResNet-50 overfits relative to ResNet-18 on 447 COVID X-rays and clusters images by hospital. Lung masking removes these shortcuts.
- Shallow beats deep on tiny data. Savioli et al. find a shallow AlexNet beats InceptionV4 on 25 ultrasound sequences.
Together, these papers show that the right capacity depends on the data, but each picks it by trying alternatives.
Two methods supply the base our adapter falls back to.
- LP+text. Few-shot Adaptation of Medical Vision-Language Models anchors a linear probe on frozen text embeddings, which are the vector representations the model assigns to each class name. It beats prompt tuning in 1-shot histology.
- SS-Text+. Silva-Rodríguez et al. make this closed-form and validation-free, with per-class adaptive regularization. Their argument that a validation split hurts more than it helps is the constraint we adopt.
Both papers fix trainable capacity at the linear head. We keep their head and their validation-free ethos, but allow internal capacity when, and only where, the data justify it.
Three capacity-measurement papers are the closest methodological relatives, and each differs from ours.
- RMT pruning. Marchenko–Pastur pruning applies the same random-matrix bulk test to trained weight matrices on ImageNet-scale models, removing noise after training. We apply the test to the update that a handful of examples request, before training, to decide what to add.
- Dominating data. Mei et al. bound required capacity by a count of "dominating points", representative cluster centers in the data. Their count depends on a hand-set distance threshold and was validated only on MNIST, CIFAR-10 and SVHN. Our threshold is set by sample size and layer shape.
- MDL ranking. Small Data, Big Decisions ranks architectures on small subsets using temperature calibration and minimum description length (MDL), the idea that the best model is the one that compresses the data most. It argues that larger models generalize better even at two examples per class. This appears to contradict the medical findings.
We reconcile the two views by separating frozen backbone size from trainable degrees of freedom. A large calibrated backbone can be fine, while the number of directions we let it move must still be identified by the data. We test this directly by crossing backbone size with the selected rank.
Research Program
The method has three steps per task.
- Fit the text-anchored probe in closed form, as in SS-Text+.
- For every attention and MLP projection in the vision encoder, compute per-example gradients on the full support set. We estimate $\sigma_l$ after whitening the gradients by their per-row and per-column scale, because raw gradients are not isotropic. We then count spikes above $\lambda_+^{(l)}$.
- Attach LoRA adapters of rank $r_l^*$ only to layers with $r_l^*>0$, initialized along the detected singular directions. Train them with a single fixed schedule. The learning rate, step count and weight decay are chosen once on a development dataset that is never a target, so no target data is spent on tuning.
The analysis program asks three questions.
- Does $r_l^*$ track the empirical crossover? We sweep shots and compare the predicted rank against the hindsight-optimal fixed rank.
- Is the analytic edge well calibrated? We compare it against an empirical null obtained by permuting labels and recomputing spikes. Anisotropy in real gradients is the main way the analytic edge could fail.
- What do the detected directions encode? On CORDA we project the top spike directions onto hospital-of-origin labels. This lets us ask whether data-supported directions can still be shortcuts, and whether the lung masking of Tartaglione et al. removes those spikes while keeping pathology-aligned ones. This is a genuine limit of the method: a test of statistical coherence cannot tell signal from a consistent confounder.
Baselines
Do-nothing reference: zero-shot classification and standard linear probing. Zero-shot classification uses the frozen text prompts. Standard linear probing fits a logistic classifier on frozen features with default regularization. These anchor the low end. In 1-shot NCT-CRC histology with a Quilt-1M backbone, LP+text reports 52.05% accuracy for standard linear probing.
Strongest low-shot competitor: LP+text and SS-Text+. LP+text reports 55.6% 1-shot NCT-CRC accuracy, against 46.05% for CoOp. SS-Text+ reports holding performance in imbalanced ophthalmology settings where linear probing drops nearly 5 accuracy points. No exact SS-Text+ scalar for these benchmarks appears in our file. These are the methods to beat because they already avoid validation splits and already win at the shot counts we target. We run them with their published defaults on identical support sets and seeds, and on the same backbones (Quilt-1M, FLAIR, MedCLIP). Since our method reduces to this head when no spikes are detected, beating them means a significant gain at 8–16 shots, or on the full prognosis cohort, without losing more than seed noise at 1–4 shots.
Strongest high-capacity competitor: fixed-rank LoRA. We run LoRA at ranks 2, 4, 8 and 16 on all layers, following the setup of Ruffini et al., and report two versions. The honest version picks the rank by a held-out split taken from the support set. The oracle version picks the best rank on the test set. No reported scalar MCC or PR-AUC for LoRA on these prognosis cohorts appears in our file. We will therefore re-run their protocol, and the reproduced numbers serve as the reference. Beating the honest version and coming within seed noise of the oracle is the success criterion.
MDL baseline: calibrated description-length selection. Following Small Data, Big Decisions, we choose among the same rank ladder by temperature-calibrated MDL computed on the support set.
Ablations isolating the spectral count. All ablations use the same gradient computation and training schedule.
- A rank rule driven by sample size alone, $r \propto \sqrt{N}$, with no spectrum. This asks whether the spectrum adds anything beyond counting examples.
- Our selected total rank redistributed uniformly across layers. This asks whether the layer placement matters.
- Detected spikes kept, but adapters initialized randomly rather than along the detected directions.
- The analytic edge swapped for the label-permutation edge.
Why not just…?
Why not just always use SS-Text+ or LP+text? At the lowest shot counts this may be exactly right, and our rule is designed to agree. The evidence that it is not always right comes from Ruffini et al., where parameter-efficient fine-tuning overtakes probing on the full Stony Brook cohort. LP+text itself notes that a frozen encoder saturates when features are misaligned with the target. The experiment that settles it is the shot sweep. If $r^*$ is zero and probing is optimal at every shot count on every benchmark, the honest conclusion is that a closed-form probe suffices. Our contribution then shrinks to a validated stopping rule, and we would report it that way.
Why not just choose the rank by leave-one-out or k-fold cross-validation, which reuses all data? This is the most serious threat. Cross-validation imposes no permanent validation tax. With a linear head it is nearly free. With LoRA it costs one training run per fold per rank. At 2–8 shots per class each fold's estimate is extremely noisy, which is the instability Ruffini et al. report for adapters. We include five-fold cross-validated rank selection as a baseline at matched total compute. If it matches our rule at 16–32 shots, the case for the spectral rule rests on its cost and its stability at lower shot counts, and we will say so.
Why not just use a fixed small rank with heavy weight decay or early stopping? Strong regularization can make an over-sized adapter behave like a small one. If so, rank is the wrong lever. We test this with fixed rank 8 under a grid of weight decay and step counts, tuned on the same non-target development dataset our method uses. If that tuning transfers across tasks as well as our rank rule does, regularization explains the gain. The layer-placement and spike-direction ablations then show whether anything specific to the spectrum remains.
Why not just train a shallow CNN from scratch? Tartaglione et al. report only 0.54 sensitivity at 0.80 specificity for an RSNA-pretrained ResNet-18 on CORDA. This suggests a low ceiling for small CNNs. However, Ruffini et al. find ImageNet CNNs with full fine-tuning the most dependable option on small imbalanced prognosis sets. We therefore include ResNet-18 with full fine-tuning as a baseline on the prognosis cohorts and on CORDA, using the EIDOSLAB codebase.
Proposed Experiments
Few-shot classification benchmarks. We use the benchmarks of LP+text and SS-Text+:
- histology: NCT-CRC, SICAPv2 and SkinCancer with Quilt-1M;
- ophthalmology: MESSIDOR, FIVES and ODIR200x3 with FLAIR;
- radiology: CheXpert, MIMIC and RSNA with MedCLIP.
Each runs at 1, 2, 4, 8 and 16 shots, with both balanced and imbalanced support sets and ten seeds. We report accuracy, with balanced accuracy for imbalanced sets.
Prognosis cohorts. We use the chest X-ray cohorts of Ruffini et al. at $K$ of 2 through 32 and on NY_all, reporting Matthews correlation coefficient (MCC) and PR-AUC. These cohorts are where the probe-to-adapter crossover has actually been observed.
Shortcut test. CORDA, with and without lung masking, serves as the shortcut test.
Decisive checks. Three outcomes settle the claim.
- Crossover prediction. The shot count at which $r^*$ first becomes positive should fall within one shot level of the empirical point where the oracle fixed-rank LoRA first beats LP+text. A systematic miss in either direction falsifies the claim that the count tracks the crossover.
- Edge calibration. The analytic and permutation edges should agree within a factor small enough not to change $r^*$ on most layers. If they disagree badly, the method should run on the permutation edge, and the random-matrix framing becomes a heuristic rather than an explanation.
- Masking and spikes. On CORDA, masking should reduce the spike directions correlated with hospital of origin while preserving accuracy-relevant ones. If spikes persist and still encode hospital, the selection rule measures coherence rather than validity. Any clinical use would then require pairing it with anatomical priors, which is a limitation we expect to state rather than hide.
References
- Benchmarking foundation models and parameter-efficient fine-tuning for prognosis prediction in medical imaging
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- Temporal Convolution Networks for Real-Time Abdominal Fetal Aorta Analysis with Ultrasound
- Few-Shot Adaptation of Medical Vision-Language Models
- Few-Shot, Now for Real: Medical VLMs Adaptation Without Balanced Sets or Validation
- Pruning Deep Neural Networks via a Combination of the Marchenko-Pastur Distribution and Regularization
- Towards Bridging Sample Complexity and Model Capacity
- Small Data, Big Decisions: Model Selection in the Small-Data Regime