Many medical image classifiers can be improved without retraining the network. One way is to replace the final linear layer with a Gaussian classifier: fit a mean and covariance (the spread and correlations of feature values) for each diagnosis in the network's feature space, then assign a new image to the diagnosis under which it is most probable. Two things break this on medical data. Deep features are skewed rather than bell-shaped, and rare diseases contribute only a few dozen images. A covariance estimated from so few examples looks far tighter than the disease really is. New patients with the rare disease then land "outside" their own class and get assigned to a common one. The first problem has a known fix: a power transform (for example, a square root applied to each feature) that makes features more Gaussian. The second problem is usually patched with hand-tuned shrinkage.
We propose to treat the second problem as what it statistically is: uncertainty about the covariance, which grows as the class gets smaller. We will fit a hierarchical Bayesian Gaussian classifier on power-transformed features. Each class's covariance is drawn toward a pooled covariance with a strength set by that class's sample count, and the classifier scores test points with the resulting heavier-tailed (Student-t) predictive distribution instead of a plug-in Gaussian. The prior strength is fit by marginal likelihood (how well the model explains the training features) rather than on a validation set, which matters because rare classes have almost no validation data. The question is whether this one count-aware correction, applied post hoc to frozen features, recovers rare-disease recall that currently requires end-to-end long-tailed training.
Core Claim
A Gaussian classifier's tail-class failures have two separable causes. One is shape mismatch: skewed, heavy-tailed features. The other is estimation error: covariances fit from tiny samples, an error that grows as the sample count shrinks. Prior work fixes the first and treats the second with ad hoc devices. Shape mismatch is addressed by Tukey de-skewing in FeCAM and Free Lunch for Few-shot Learning: Distribution Calibration, and by unit-sphere projection in Mahalanobis++. FeCAM handles small samples with a fixed shrinkage constant. Distribution calibration borrows covariance from the semantically nearest classes. Mahalanobis++ assumes a single covariance shared by all classes.
We claim that a Normal-inverse-Wishart (NIW) posterior predictive handles both shared and class-specific structure, with the balance set by the data. NIW is the standard conjugate prior for a Gaussian's mean and covariance. Its posterior predictive interpolates automatically between a shared covariance (as class size goes to zero) and the class's own covariance (as class size grows), and it inflates distances for small classes by exactly the amount sampling uncertainty warrants.
Concretely, let $z$ be a feature vector after the per-dimension power transform and a dimensionality reduction to $d$ dimensions. Let $\Sigma_0$ be the pooled within-class covariance. For class $c$, let $n_c$ be the number of training images, $\bar z_c$ their mean and $S_c$ their scatter matrix. With a prior of strength $\nu_0$ and a weak prior on the mean, the class score is a multivariate Student-t:
$$p(z \mid c) = t_{\nu_c}\!\left(z \;\middle|\; \bar z_c,\; \frac{(n_c+1)\,\Psi_c}{n_c\,\nu_c}\right), \qquad \Psi_c = \nu_0 \Sigma_0 + S_c,\quad \nu_c = \nu_0 + n_c - d + 1.$$
Here $\nu_c$ is the degrees of freedom: small for rare classes, which gives them heavy tails and wide acceptance regions, and large for common classes, which approach an ordinary Gaussian. The single hyperparameter $\nu_0$ is chosen to maximize the closed-form NIW marginal likelihood of the training features. Prediction uses a uniform class prior, because the target is balanced accuracy rather than raw accuracy.
What is new relative to the closest work is that small-sample correction is derived rather than tuned. The prior is pooled across all classes, not borrowed from semantic neighbors. Gaps 2 and 4 of the file explain why borrowing is unsafe in medicine: a common benign lesion need not share variance structure with a rare malignant one.
Research Program
The program has three parts.
Diagnosis. On frozen features from several backbones, we will measure how far each class's held-out samples deviate from the model's predicted distance distribution. The backbones are an ImageNet ResNet-50, the CLIP image encoder used by TFA-LT, and ERM- and GCL-trained backbones from MONICA. Under a correct model, squared Mahalanobis distances of held-out class members follow a known scaled-F distribution. The fraction of a class's test points falling inside the model's 95% region is therefore a direct, per-class measure of miscalibration. We expect plug-in Gaussians to under-cover badly for small $n_c$ and the Student-t predictive to close this gap.
We will also check whether a power transform fitted on pooled training features, which are dominated by head classes, actually Gaussianizes tail classes. This is untested; the file's evidence for de-skewing (Leveraging the Feature Distribution in Transfer-based Few-Shot Learning) comes from balanced natural-image benchmarks. The comparison is per-class normality before and after transformation, plotted against $n_c$.
The classifier. We will build the classifier above and compare three feature maps feeding it: Tukey square root, a Yeo-Johnson transform fitted per dimension, and $\ell_2$ normalization. The dimensionality reduction (PCA to $d \in \{64, 128, 256\}$) is required because the NIW predictive needs $\nu_0 > d-1$. We treat $d$ as a reported sensitivity, not a tuned knob.
Multimodality. Medical classes are often mixtures of subtypes or acquisition sites, the concern raised by FedNPR and DuBE. We will test this as a bounded extension: a mixture of $K$ NIW components per class, with components sharing the same prior, used only for classes with enough data to support it. This tells us whether residual errors come from unimodality or from estimation error, without making the proposal depend on mixtures.
Baselines
The naive reference is a linear probe trained with plain cross-entropy on the same frozen features. It reproduces the ERM regime that MONICA reports collapsing to 12.67% tail accuracy on ISIC-2019-LT at imbalance ratio 500.
The strongest cheap competitor on frozen features is a logit-adjusted or Balanced Softmax linear probe. Balanced Softmax is the margin correction FedNPR uses. Like our method, it corrects for class frequency at no extra cost, so beating it is the minimum bar.
The strongest existing distance-based method is FeCAM: Tukey transform, class-specific Mahalanobis distance, fixed covariance shrinkage and correlation normalization. FeCAM reports 70.8% on 10-task CIFAR-100 but has no reported medical result. We will run its published recipe on our features, with its shrinkage constants tuned on the same validation split and with the same grid budget we allow every baseline. We will also run Mahalanobis++ (unit-norm features with one shared covariance) and the distribution-calibration recipe of Free Lunch (top-$k$ neighbor covariance borrowing, then sampling features to train a logistic regression).
The strongest end-to-end reference is GCL as reported in MONICA, at 71.33% tail accuracy on ISIC-2019-LT at $r=500$. ProCo reports accuracy 0.887 and F1 0.763 on ISIC2018, and accuracy 0.837 and F1 0.674 on APTOS2019. We will apply our classifier on top of a GCL-trained backbone, so the comparison asks whether a post-hoc head adds anything beyond the best training recipe, not whether it replaces it.
Two ablations isolate the key ingredient. The first keeps our NIW posterior covariance but scores with a plug-in Gaussian instead of the Student-t; this removes the count-dependent tail inflation. The second sets $\nu_0$ to a constant across datasets instead of fitting it by marginal likelihood. A third ablation removes the power transform.
All methods see identical features, splits and seeds. Beating a baseline means a higher balanced accuracy and higher accuracy on the rarest third of classes, with bootstrap 95% intervals over test images and three backbone seeds that exclude zero. Overall accuracy must not drop by more than one point.
Why not just…?
Why not just use a Balanced Softmax or logit-adjusted linear probe? It fixes the decision threshold between classes but cannot model that a rare class's features are spread differently from a common one's. A linear probe also learns tail weights from the same few points that make tail covariances unreliable, which is the instability FedNPR documents for classifier weights under imbalance. This alternative is a real threat on mildly imbalanced data such as APTOS2019, where the imbalance ratio is 10. We expect our advantage, if any, to grow with imbalance ratio, and the MONICA ISIC-2019-LT ratios of 100, 200 and 500 test exactly that trend.
Why not just use one shared covariance, as Mahalanobis++ and linear discriminant analysis do? A shared covariance avoids estimating tail covariances at all, and it is the limiting case of our model as $\nu_0$ grows large. If the marginal likelihood drives $\nu_0$ to that limit, and shared-covariance scoring matches us on tail accuracy, then class-specific structure does not matter for these datasets and we will report that. The fitted $\nu_0$ and the per-class coverage curves settle this directly.
Why not just use FeCAM with well-tuned shrinkage? With enough validation data, a tuned constant shrinkage may approximate our count-dependent one. In medical tails, however, the validation split may contain a handful of images per rare class, so tuning is noisy. The decisive comparison is FeCAM tuned on the real, small validation split versus FeCAM tuned by an oracle on the test set, both set against our evidence-fitted model. If the oracle-tuned FeCAM matches us and the realistic one does not, the contribution is removing that tuning dependence, not accuracy per se.
Why not just train end to end with GCL or ProCo? These remain the strongest systems, and our method is not a replacement. The experiment on top of the GCL backbone tells us whether count-aware Gaussian scoring corrects residual tail errors that contrastive training leaves behind. If it does not, the proposal's value narrows to cheap adaptation of frozen foundation features, as in the TFA-LT setting.
Proposed Experiments
The main benchmark is the MONICA suite. We use ISIC-2019-LT at imbalance ratios 100, 200 and 500, plus DermaMNIST, OrganAMNIST, PathMNIST and KVASIR, together with ISIC2018 (imbalance ratio 58) and APTOS2019 under the 7:3 splits used by ProCo, so its numbers are directly comparable. Metrics are balanced accuracy, macro-F1, tail-third accuracy, expected calibration error, and the per-class coverage gap. Because MONICA reports that long-tailed methods improve out-of-distribution detection on OrganAMNIST, we will also report FPR@95 (the false-positive rate on out-of-distribution images at 95% true-positive rate) for detecting held-out OrganAMNIST views. Here the Student-t scores replace Mahalanobis scores. No reported result was found on Lacuna for a distance-based detector on this benchmark.
Three outcomes would decide the proposal. The first is a coverage gap: plug-in Gaussian coverage of tail classes should fall well below 95% and scale with $1/n_c$, and the Student-t predictive should restore it. If plug-in coverage is already fine, the estimation-error story is wrong. The second is the ablation result: replacing the Student-t with a plug-in Gaussian, while holding the posterior covariance fixed, should account for most of the tail-accuracy gain. The third is the imbalance-ratio trend: the margin over Balanced Softmax and FeCAM should widen from APTOS2019 to ISIC-2019-LT at ratio 500. A flat or reversed trend would indicate that count awareness is not what limits tail performance.
References
- FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning
- Free Lunch for Few-shot Learning: Distribution Calibration
- Mahalanobis++: Improving OOD Detection via Feature Normalization
- Text-Guided Foundation Model Adaptation for Long-Tailed Medical Image Classification
- MONICA: Benchmarking on Long-tailed Medical Image Classification
- Leveraging the Feature Distribution in Transfer-Based Few-Shot Learning
- Non-parametric regularization for class imbalance federated medical image classification
- Towards Inter-class and Intra-class Imbalance in Class-imbalanced Learning
- ProCo: Prototype-Aware Contrastive Learning for Long-Tailed Medical Image Classification