Deciding How Much to Share: Pilot-Estimated Covariance Heterogeneity for Few-Shot Gaussian Classifiers

Classifiers built on frozen deep features, such as CLIP image embeddings, a frozen ResNet, or a graph network over molecules, increasingly work in a simple way. They fit a Gaussian to each class and classify by distance that accounts for the shape of the feature cloud. With only a handful of examples per class, no one can estimate each class's own covariance matrix, meaning the shape of its spread across feature dimensions. So most methods assume every class shares one matrix. That assumption holds up well in some settings, where it gives a "hard-to-beat" training-free baseline for CLIP. In others it clearly breaks: in continual learning, new classes spread out in visibly different ways. Current methods pick a side by fiat, either always sharing or always estimating per-class shapes with a hand-set blending weight.

This proposal treats the choice as a question that small data can answer. The approach borrows the logic of a clinical pilot study, which runs a small preliminary study to estimate a variance component before committing to the main one. Here the variance component is how much class covariances actually differ from the pooled one. We estimate this single heterogeneity quantity by pooling evidence across all classes, and optionally across earlier tasks. We then subtract the part that is only sampling noise and use what remains to set, in closed form, how far each class's covariance should move away from the shared matrix given its sample count. With one or two shots the method reduces to pure sharing. As data accumulates, it moves toward class-specific shapes, but only at the rate the measured heterogeneity justifies.

Core Claim

Consider a covariance estimate built from very few examples. Its error has two sources: true between-class heterogeneity and estimation noise. The noise part has a known expected size under a Gaussian model, so it can be subtracted from the observed dispersion of class covariances around the pooled one. What remains is an estimate of real heterogeneity. It is stable because it pools across classes, even when no single class has enough data to estimate its own. Let $\hat\Sigma_c$ be class $c$'s sample covariance from $N_c$ examples and $\bar\Sigma$ the pooled shared covariance. Let $\nu = \mathrm{tr}(\bar\Sigma^2) + \mathrm{tr}(\bar\Sigma)^2$, which is the Gaussian expected squared Frobenius error of a covariance estimate, per degree of freedom. We estimate heterogeneity by the method of moments and set the per-class blending weight as follows:

$$\hat\tau^2 = \max\Big(0,\ \tfrac{1}{C}\textstyle\sum_c \|\hat\Sigma_c - \bar\Sigma\|_F^2 - \tfrac{1}{C}\sum_c \tfrac{\nu}{N_c-1}\Big), \qquad \lambda_c = \frac{\hat\tau^2}{\hat\tau^2 + \nu/(N_c-1)}.$$

The regularized class covariance is then $\lambda_c \hat\Sigma_c + (1-\lambda_c)\Sigma_{\text{shared}}$. Here $\Sigma_{\text{shared}}$ is the pooled matrix shrunk toward a scaled identity, exactly as in the training-free CLIP baseline. The form of $\lambda_c$ is the standard empirical-Bayes signal-to-noise ratio: signal variance divided by signal plus noise variance.

We also claim that $\hat\tau^2$ is a useful diagnostic on its own. It should be near zero on the backbones and datasets where shared-covariance methods win, and clearly positive where heterogeneous methods win. This would reconcile two findings that currently point in opposite directions.

Relation to Prior Work

Several papers motivate the shared side by showing that pooling is strong at low shot. A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation fits Gaussian discriminant analysis (a classifier with one Gaussian per class) using a shared, identity-shrunk covariance on frozen CLIP features, and ensembles it with the text classifier. Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis keeps one running shared covariance so that new classes do not overwrite old ones. The test-time methods ADAPT (Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment), FreeTTA (Free on the Fly) and OGA (Online Gaussian Test-Time Adaptation of Vision-Language Models) all tie covariance across classes, either during closed-form updates or in online expectation-maximization on unlabeled streams. None of these papers asks whether the tie is warranted on a given task. We keep their shared estimator as our shrinkage target and add a data-driven exit from it.

Other papers constrain us, because they show that sharing can be wrong. FeCAM argues that frozen features give novel classes heterogeneous, anisotropic spreads. It models per-class covariances after a power transform, with shrinkage toward identity rather than toward the pooled matrix. Adaptive Dimension Reduction and Variational Inference for Transductive Few-Shot Classification reports that per-class covariances overfit badly at low shot. Our estimator is designed to respect both findings: it can only be heterogeneous where the pooled evidence says heterogeneity exceeds noise.

The closest method is Simple CNAPS (Improved Few-Shot Visual Classification), extended in Beyond Simple Meta-Learning. It already blends each class covariance with the task-pooled one, but its weight is a fixed monotone function of $N_c$ that ignores the data. We keep its functional form and replace the heuristic weight with one estimated from data. Under our weight, two tasks with identical shot counts but different true heterogeneity receive different blends; under Simple CNAPS they receive the same one. A Strong Baseline for Molecular Few-Shot Learning shows that quadratic (Mahalanobis) probing on frozen graph network embeddings beats meta-learning, provided the covariance is regularized against degenerate solutions. That paper supplies our third domain.

Research Program

The work has three parts.

First, we build the estimator. The noise correction uses the Gaussian formula above plus a kurtosis-corrected variant, because deep features are not Gaussian and heavy tails inflate the true noise. All quantities cost $O(CD^2)$, with no matrix inversions beyond the one the base classifier already performs.

Second, we build a cross-task "pilot" variant. When a new task has few classes, $\hat\tau^2$ from that task alone is noisy. We therefore estimate heterogeneity on base or earlier-task data, expressed as a ratio to $\nu$, and use it as a prior that the new task's own estimate updates. This step is the literal pilot-study analogue: a cheap early sample estimates the variance component that sizes the later decision.

Third, we build a streaming variant for continual learning and test-time adaptation. Here Deep SLDA's running statistics are augmented with per-class running covariances, and $\lambda_c$ is recomputed as each class accrues data. Sharing then becomes a gate that opens gradually and differently per class.

Alongside these, we evaluate $\hat\tau^2$ as a diagnostic by checking whether it predicts, before any labels are spent on validation, whether shared or heterogeneous classifiers win on a dataset.

Baselines

The naive reference points are the nearest-class-mean Euclidean classifier and, for CLIP, the zero-shot text classifier. Any covariance method that fails to beat these is not worth discussing.

The strongest shared competitor is the training-free Gaussian discriminant classifier of A Hard-to-Beat Baseline. It is the right opponent because our method contains it as the special case $\lambda_c = 0$, so any gain must come from the heterogeneity term. We rerun it with its published shrinkage and ensembling on identical features and splits, and our method uses the same text ensembling. I did not find a reported accuracy figure in the Lacuna file for this method; the file notes that its tables were not retrieved, so the comparison numbers will come from our own reproduction.

The strongest heterogeneous competitor is FeCAM. We run it with its power transform and shrinkage on the same frozen continual-learning features and apply the same power transform to our method, so the transform is not a confound. No reported result was found on Lacuna for FeCAM.

The closest structural baseline is Simple CNAPS-style hierarchical shrinkage with its heuristic $\lambda_c$. It is dropped into the identical pipeline so that the only difference is how $\lambda_c$ is chosen. No reported result was found on Lacuna for Simple CNAPS.

Two further baselines are necessary. The first is per-class Ledoit–Wolf-style shrinkage toward the pooled matrix, where each class estimates its own intensity. The second is a globally cross-validated $\lambda$. Both receive the same labeled data. The cross-validation baseline receives every label we use, including any pilot split.

Two ablations isolate the key ingredient. One removes the noise subtraction, using raw dispersion. The other removes cross-class pooling, estimating $\tau^2$ per class.

Beating the baselines means the following. The method should match the shared classifier within noise at 1–2 shots and on datasets where $\hat\tau^2$ is near zero. It should give a statistically significant gain over both the shared classifier and Simple CNAPS at 8–16 shots and in continual settings. It should also never fall below the better of shared and FeCAM by more than seed-to-seed variation. Robustness across regimes, without per-dataset tuning, is the real target.

Why not just…?

Why not just cross-validate a single blending weight on held-out labels? At 1–4 shots, a held-out split either consumes most of the support set or is too small to rank candidate weights reliably. Leave-one-out also destroys class covariances when $N_c \le 2$. This is still a genuine threat at 16 shots. We give cross-validation every label our method sees. If it matches us there, we will report that our estimator's advantage is confined to the low-shot and streaming regimes, where cross-validation is not available at all.

Why not just use Ledoit–Wolf shrinkage per class toward the pooled target? This is the most serious alternative, because it is almost our estimator. The difference is that it estimates shrinkage intensity from each class's own few samples. With two to five examples that intensity estimate is itself very noisy, and with one example it is undefined. Pooling the heterogeneity estimate across classes is exactly the step that addresses this. The no-pooling ablation is this baseline in disguise, and if it ties the full method across shot counts, our main claim fails.

Why not just always share, since sharing is already hard to beat? On CLIP it may be the right call, and our method is built to recover it there. However, FeCAM's evidence of heterogeneous novel-class spreads in continual learning suggests that permanent sharing leaves accuracy on the table once classes accumulate data. The streaming experiment tests whether the gate opens in those settings and whether opening it helps.

Proposed Experiments

We cover three domains, each using the evaluation suites of the cited papers.

The first is few-shot CLIP adaptation, using the dataset collection and 1/2/4/8/16-shot protocol of the training-free Gaussian discriminant baseline. The second is exemplar-free class-incremental learning on frozen backbones, following the FeCAM and Deep SLDA setups. The third is molecular few-shot prediction with frozen graph network embeddings, following A Strong Baseline for Molecular Few-Shot Learning. Metrics are top-1 accuracy (or the molecular paper's metric), average incremental accuracy for continual learning, and expected calibration error, all over at least five seeds with confidence intervals.

The first decisive check is synthetic. We draw Gaussian classes with controlled true heterogeneity and verify that $\hat\tau^2$ tracks the truth and that $\lambda_c$ approaches the oracle weight as $N_c$ grows. The second check is the diagnostic test. Across all datasets and backbones, we measure whether $\hat\tau^2$ correctly predicts the sign of the accuracy gap between FeCAM and the shared classifier. We will report rank correlation and treat a correlation near zero as falsifying the diagnostic claim. The third check is the no-pooling and no-noise-subtraction ablations at 2–4 shots, which determine whether the pilot-style pooled variance estimate is doing the work. The fourth is the cross-task pilot variant on tasks with few classes, compared against the within-task estimate.