Estimation-Aware Quadratic Weighting of Frozen Models for Few-Shot Transfer

When a practitioner has a handful of labeled images from a new domain, say five chest X-rays per disease, and a shelf of pretrained networks, the standard move is to pick one network and fit a small classifier on its features. A better move is to combine several networks with weights. Choosing those weights by minimizing a squared-error objective is attractive because the problem becomes a convex quadratic program (QP), an optimization with a single global optimum that standard solvers find exactly. The weakness is the one this note targets. The coefficients of that quadratic objective are themselves estimated from the same five examples per class, so the "globally optimal" weights are optimal for a very noisy objective, and in practice they often overfit.

The proposal separates the objective into two parts. The first says how much the models' predictions overlap with one another. It does not need labels, so we estimate it from the whole unlabeled query pool, which contains hundreds of points. The second says how well each model's predictions agree with the true labels. It needs labels, so we estimate it only from held-out predictions on the support set (the few labeled examples), and we shrink it toward "no information" by an amount computed from its own sampling variance. The result is still one small QP per task, with no hyperparameter search. When the labels carry little signal, the weights fall back to uniform averaging automatically. When they carry a lot, the weights concentrate on the models that actually transfer.

Core Claim

The claim is narrow. On few-shot tasks with a large domain shift, the main source of error in quadratic model-combination is variance in the estimated objective, not the choice of objective. We expect that estimating the label-free part transductively and shrinking the label-dependent part analytically will beat three alternatives at 1 to 5 shots: the unshrunk QP computed on the support set alone, uniform averaging, and concatenating all features into one regularized linear classifier. We expect the advantage to be largest on the most distant domains, ChestX and ISIC.

Concretely, suppose there are $K$ frozen backbones. For each backbone $k$ we fit a closed-form ridge classifier on the support set. We then compute its leave-one-out class-score vector $s_k(x_i)$ for each support point $x_i$; for ridge regression this is exact and cheap through the hat matrix. We also compute its ordinary score vector $\tilde s_k(x)$ on unlabeled query points. Weights $w$ on the simplex solve

$$ \min_{w \in \Delta_K} \; w^\top A\, w \;-\; 2\,(1-\lambda)\, \hat b^\top w \;+\; \lambda\, \| w - u \|^2 , $$

The symbols are defined as follows:

  • $A_{jk}$ is the average inner product of $\tilde s_j$ and $\tilde s_k$ over the query pool. It is the label-free "overlap" term, playing the role of the ambiguity and disagreement quantities in Heskes' logarithmic opinion pools and the second margin moment in MinCq.
  • $\hat b_k$ is the average inner product between the leave-one-out scores $s_k(x_i)$ and the one-hot labels $y_i$ over the support set.
  • $u$ is the uniform weight vector.
  • $\lambda \in [0,1]$ is set in closed form: the estimated sampling variance of $\hat b$ (its per-point empirical variance divided by the number of support points), relative to the spread of $\hat b$ across models. This is a James–Stein-style rule, chosen because $\hat b$ is a sample mean whose noise we can measure directly.

The 1-shot case needs special handling, because leaving out the only example of a class destroys that class. There we compute held-out scores against augmented views of the left-out image instead. We report this regime separately, since it is where the proposal is most likely to reduce to uniform averaging.

Relation to Prior Work

The closest work is H-ensemble. It linearly combines frozen source feature extractors for few-shot, source-free transfer, shows that the negative H-score is convex quadratic in the weights, and reports 89.86% 10-shot accuracy on VisDA-2017. H-ensemble motivates and constrains this proposal, and we differ from it in three ways:

  • It estimates every term of its quadratic objective from the labeled support set.
  • It combines features rather than held-out predictions.
  • It was evaluated only on natural-image benchmarks.

We keep its convex, backprop-free spirit and change what each term is estimated from.

The PAC-Bayes line supplies the decomposition we exploit. MinCq reduces C-bound minimization to a QP over the first and second margin moments. Its transductive variant already estimates the second moment on unlabeled data, but it gave inconsistent gains over transductive SVMs, and MinCq depends on a margin parameter tuned by cross-validation. Self-bounding C-bound minimization tightens the certificates, with a mean bound of 0.378 versus 0.925 for MinCq. It was run only with decision trees, and its bounds loosen at few-shot sizes. We borrow the insight that the disagreement term needs no labels. We replace the tuned margin parameter with a variance-derived shrinkage, and we test on deep features, which neither paper did.

SAFER motivates our "never much worse than a safe default" design. It projects a supervised baseline onto the convex hull of semi-supervised regressors through a QP and proves non-degradation for regression. We do not claim its guarantee for classification. Our analogue is weaker and empirical: shrinkage toward uniform bounds how far a noisy objective can pull the weights.

QRLP-Boost shows that a quadratic regularizer lets totally corrective weights be solved in one step, rather than through iterative search. Our $\lambda$ term is in the same spirit, but its strength is set by an estimate of the objective's noise rather than by hand.

Two findings frame the regime. The BSCD-FSL benchmark shows that on ChestX, meta-learners do worse than random weights, and that plain pretrain-and-fine-tune beats meta-learning by 12.8% on average. The cross-domain classifier comparison finds that L2-regularized one-vs-rest logistic regression and cosine classifiers on frozen features are the strongest heads. The same paper finds that cross-validated hyperparameters on tiny support sets often underperform defaults. That second finding is the direct motivation for avoiding any tuning loop.

Research Program

We will build a library of roughly eight to twelve frozen backbones: ImageNet-supervised and self-supervised convolutional and transformer models, image-text models, and at least one model pretrained on medical images. Where cheap, we also include penultimate and intermediate layers as separate "models." Every model gets the same closed-form ridge head. Its leave-one-out and query scores feed the QP above, which a projected-gradient or off-the-shelf QP solver handles in milliseconds for $K \le 50$.

The analysis has three parts:

  1. Variance decomposition. On tasks where we hold out large labeled test sets, we compute the "oracle" $A$ and $b$ from the test labels. We then measure how much of the weight error, and of the accuracy gap to the oracle weights, comes from noise in $\hat b$ versus noise in the support-set version of $A$.
  2. Shrinkage rule. We check whether the analytic $\lambda$ tracks the $\lambda$ that is best in hindsight across shots and domains.
  3. Heterogeneous libraries. We study behavior as $K$ grows. The open question is whether the simplex constraint plus shrinkage keeps weights sparse and stable, or whether redundant near-duplicate models dilute them. This is the concern CLDFD addresses with a hand-tuned top-$h$ feature truncation.

Baselines

The do-nothing reference is uniform averaging of the $K$ ridge heads' scores. It needs no estimation at all. H-ensemble argues that it gives too much weight to irrelevant sources, and at 1 shot it is the natural floor. We count ourselves as beating it only if our mean accuracy over 600 tasks per domain and shot exceeds it with a paired confidence interval excluding zero on at least ChestX and ISIC.

The strongest single-model competitor is the best frozen backbone with L2-regularized logistic regression, which the classifier comparison identifies as a top head. We will run it two ways. One chooses the backbone by support-set cross-validation, which is honest. The other chooses the backbone per domain using test labels, which is an upper bound. The same backbone library, support sets and augmentation apply to both. That paper reports no absolute per-dataset numbers; no reported result found on Lacuna for its per-domain accuracies.

The strongest existing combination method is H-ensemble, run on our library with its analytic head. For multi-source adaptation of representations we compare against LAMR++ from Task-Adaptive Multi-Source Representations, which reports 73.07% average accuracy across BSCD-FSL and outperforms TSA by 3.2%. Its reported number uses its own backbones, so we treat that comparison as indicative only. The fair comparison is H-ensemble on identical features. For EuroSAT 1-shot specifically, CLDFD reports 82.52%. BSCD-FSL's absolute ChestX accuracies are not in the file; no reported result found on Lacuna for ChestX absolute accuracy of the strongest method.

Two ablations isolate the key ingredient:

  • The same QP with $A$ computed on the support set only, which removes the transductive estimate.
  • The same QP with $\lambda=0$, which removes shrinkage.

If either ablation matches the full method, that component is not doing the work. Every method gets the same tuning budget, which is zero test-label access, except for the oracle-selected rows. Those rows are labeled as upper bounds.

Why not just…?

Why not just average all models uniformly? At 1 shot this may be as good as anything, and our method is built to reduce to it in that case. The test is whether, at 5 and 20 shots, weighting beats averaging on ChestX, where H-ensemble's argument predicts that off-domain sources hurt. The uniform-average baseline and the $\lambda$ trajectory across shots will show directly when weighting starts to pay.

Why not just concatenate every model's features and fit one L2-regularized logistic regression? This is the most serious threat. Regularized linear heads are the strongest known choice on frozen cross-domain features, and concatenation lets the regularizer do implicit model selection. Our counter-argument is that one regularization strength must then govern thousands of dimensions of very different scale and relevance, and choosing that strength is exactly the noisy cross-validation step that fails on tiny support sets. We will run concatenation both with default and with cross-validated regularization. If it matches our method at every shot, we will conclude that the QP adds nothing beyond regularized concatenation and report that.

Why not just pick the single best backbone by cross-validation? Selection is a hard, high-variance version of weighting. The classifier comparison paper already shows that small-support validation is unreliable. The gap between the cross-validated selection row and the oracle selection row measures this cost directly. We expect our weights to recover a substantial part of that gap.

Proposed Experiments

The main benchmark is BSCD-FSL (CropDiseases, EuroSAT, ISIC, ChestX), run as 5-way tasks at 1, 5 and 20 shots with 600 tasks each and 15 unlabeled queries per class. The query pool is used transductively only for $A$, never for labels. As a second medical check we will sample few-shot tasks from MedMNIST v2 (MedMNIST v2), specifically PathMNIST, DermaMNIST and OCTMNIST. We note that the 28×28 resolution removes detail, so these results are a robustness check, not a clinical claim. Metrics are mean accuracy with 95% intervals, paired differences against each baseline, weight entropy, and the correlation between analytic and hindsight-best $\lambda$.

Three checks are decisive.

  1. Where the gain comes from. The full method should beat both ablations and uniform averaging on ChestX and ISIC at 5 shots. If only the transductive $A$ matters, the story becomes "use unlabeled data for the disagreement term," which is still useful but narrower.
  2. Behavior under pure noise. When we randomly shuffle support labels, the analytic $\lambda$ should approach 1 and accuracy should approach that of uniform averaging. This tests that the shrinkage actually detects noise rather than merely regularizing.
  3. Comparison with concatenation. The comparison against regularized concatenation decides whether the quadratic combination is worth proposing at all.