Is It Ranking or Bias? Diagnosing Balanced-Accuracy Loss in Imbalanced Few-Shot Medical Adaptation

When a pretrained medical image model is adapted to a clinic's own labels with only a handful of examples, those examples are rarely balanced. Healthy cases are plentiful, and severe disease stages may have one example or none. Recent work shows that standard adapters lose several points of balanced accuracy in this setting. Balanced accuracy is the average of per-class recall, so every disease counts equally however rare it is. A method built for the imbalanced case recovers most of that loss. The field treats this as one problem, but it is really two. The adapted classifier may fail to separate rare-class images from common ones in its scores. Or it may separate them well but place its decision rule in the wrong spot, systematically favoring the classes it saw most. These failures need different remedies. The second can be fixed with a per-class additive correction to the scores, which costs almost nothing. The first needs better representations or priors.

We propose to measure the two failures separately. Balanced accuracy splits into a bias-invariant separability term and a decision-bias term. The separability term is computed with a multi-class ranking metric oriented on logit differences, so that no per-class offset can change it. We add an analytic variance for that ranking metric, so that differences between adapters can be tested across many random draws of the small labeled set. The practical question this answers is whether the gains reported by purpose-built imbalance-robust adapters come from better separation or from fixing a prior mismatch that a one-line logit correction would also fix. The answer determines where future method work should go.

Core Claim

The claim has three parts. First, under imbalanced and missing-class support sets, a large share of the balanced-accuracy drop of frozen-backbone adapters is decision bias rather than lost separability. Second, this share can be measured without labels beyond the test set, using an oracle-offset bound and a logit-oriented version of AUCμ. Third, once the bias share is known, it predicts whether cheap validation-free prior corrections will close the gap to the strongest specialized adapter. If the decomposition shows the gap is mostly separability, the cheap fixes should fail, and the specialized adapter's text prior is doing real work. Either outcome is informative, and the experiments are designed so that both are possible.

Background and Positioning

The closest work is Few-Shot, Now for Real: Medical VLMs Adaptation without Balanced Sets or Validation. It introduces SS-Text+, a training-free closed-form adapter over frozen vision-language embeddings. SS-Text+ falls back on class text descriptions when a class has few or no images, repels class prototypes apart, and is evaluated on average class accuracy under "Standard," "Relaxed," and "Realistic" (missing-class) support regimes. That paper supplies our regimes, backbones, and strongest baseline. It reports only the aggregate balanced accuracy, so it cannot say which of its mechanisms repairs which failure. The paper itself notes that it does not evaluate rank-based metrics. We do not propose a competing adapter. We propose the measurement that explains why its adapter works, plus a test of whether simpler corrections suffice. Few-shot Adaptation of Medical Vision-Language Models supplies LP+text, a text-anchored linear probe for balanced few-shot support sets. It motivates the separability side of the question: with one image per class, a purely visual probe cannot place a stable boundary, and text embeddings anchor it.

On the metric side, AUCμ: A Performance Metric for Multi-Class Machine Learning Models defines a multi-class area-under-curve that equals 1 for a perfect classifier, 0.5 for random scores, and is insensitive to class ratios. It fixes the flaw of pairwise averaging, which can score a perfect classifier as low as 0.67. Its authors leave open closed-form confidence intervals, and we take that on. We also reuse its orientation machinery in a specific way. When the pairwise orientation is the difference of logits rather than of probabilities, any per-class additive offset cancels. The resulting score measures separability alone. Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study constrains our evaluation hygiene: macro scores are arithmetic means of per-class scores over classes present in the test split. That paper also motivates the bias side. Once per-class thresholds were tuned, simple models matched complex ones, which suggests that decision placement can masquerade as model quality. Finally, NCT-CRC-HE: Not All Histopathological Datasets are Equally Useful constrains interpretation. On NCT-CRC, 48-bin color histograms alone reach 82.2% accuracy. Any separability we measure there could be color shortcutting, so we include that control.

The Decomposition

Let $z(x) \in \mathbb{R}^K$ be the adapted classifier's logits over $K$ classes. Let $\mathrm{BA}(z)$ be the balanced accuracy of predicting $\arg\max_k z_k(x)$. For a per-class offset vector $b \in \mathbb{R}^K$, define the offset-optimal balanced accuracy

$$\mathrm{BA}^\star(z) = \max_{b} \mathrm{BA}(z + b), \qquad \Delta_{\text{bias}}(z) = \mathrm{BA}^\star(z) - \mathrm{BA}(z) \ge 0.$$

Here $\Delta_{\text{bias}}$ is the balanced accuracy lost purely to where the decision rule sits. $\mathrm{BA}^\star$ depends only on how well the scores can be separated. We report $\mathrm{BA}^\star$ alongside a logit-oriented AUCμ, $\mathrm{AUC}_\mu^{\ell}$, in which the pairwise score for classes $i,j$ is $z_i - z_j$. We use two separability measures because $\mathrm{BA}^\star$ depends on the argmax geometry while $\mathrm{AUC}_\mu^{\ell}$ is threshold-free, and agreement between them guards against artifacts of either. Estimating $b$ on the same test points that score it is optimistic. We therefore cross-fit: $b$ is fit on one half of the test set and scored on the other, in both directions, with a coordinate-ascent search over $b$. The fitted $b$ is diagnostic only and is never deployed, because it uses test labels.

Research Program

The work has three pieces. The first is the decomposition applied across adapters and regimes. For each backbone, dataset, and support regime, we draw many random support sets, run every adapter, and record $\mathrm{BA}$, $\mathrm{BA}^\star$, $\Delta_{\text{bias}}$, and $\mathrm{AUC}_\mu^{\ell}$.

The second is inference. AUCμ is a generalized Mann–Whitney statistic, an average over pairs of instances from different classes. We will derive its variance by the structural-component route that gives DeLong-style intervals for binary AUC, extended to the argmax partition and to paired comparisons of two adapters on the same test set. The derivation is checked against bootstrap intervals, including at small class counts. Few-shot results have two sources of randomness: which support images were drawn, and the finite test set. We therefore report a two-level variance, between support draws and within draw. We state explicitly which comparisons in the current literature would survive it.

The third is validation-free bias correction, tested as a hypothesis rather than sold as a method. We consider two offsets that need no held-out labels. One subtracts the log of smoothed support counts. The other is derived from the adapter's own implicit prior, estimated as the mean softmax output on the support images. The prediction is that on the Relaxed regime these corrections recover most of $\Delta_{\text{bias}}$ for a plain linear probe. On the Realistic regime they cannot, because a class with zero images has no learned score direction, only whatever the text prior provides. That is where SS-Text+'s mechanism should show up as a separability gain.

Baselines

The naive reference point is the zero-shot vision-language model with class text prompts and no adaptation. It anchors how much separability the backbone already has. It also has a natural decision bias of its own, which the decomposition will expose.

The standard linear probe is logistic regression on frozen image embeddings, trained with cross-entropy on the full support set. It is the adapter that Few-Shot, Now for Real reports losing nearly 5 points of accuracy in the Realistic regime. LP+text from Few-shot Adaptation of Medical Vision-Language Models is the balanced-support text-anchored probe. The strongest reported number available for it is 55.6% 1-shot histology accuracy, against 52.05% for the plain linear probe and 46.05% for CoOp prompt tuning, all from that paper. These are plain accuracies under balanced supports, not balanced accuracy under imbalance.

The strongest competitor is SS-Text+. It was designed for exactly our regimes and eliminates the validation set. For SS-Text+ under the Realistic regime, no reported per-dataset balanced-accuracy result was found on Lacuna. The file reports only the linear probe's "nearly 5 points" drop and a greater than 15x speedup. We will therefore rerun it.

Fairness is enforced in four ways. Every method uses the same frozen embeddings from the backbones SS-Text+ used: CONCH for histology, FLAIR for ophthalmology, CONVIRT for radiology. Every method uses the same support draws and the same prompts. No method receives a validation split, and gradient-based methods use fixed published defaults, so none gets a tuning budget the closed-form method lacks. A secondary arm gives every method an equal validation split carved from the support set, to measure the validation tax directly.

The ablations that isolate the key ingredient are the linear probe with each of the two validation-free offsets, and the linear probe with the cross-fitted oracle offset. The oracle-offset probe is an upper bound on what any bias fix can achieve. Beating SS-Text+ is not the goal. The decisive result is whether the offset-corrected linear probe comes within the paired confidence interval of SS-Text+. It should do so on Relaxed supports if our bias hypothesis holds, and fall outside it on Realistic supports if text priors are what matter for missing classes.

Why not just…?

Why not just apply logit adjustment by support counts and skip the diagnosis? This is the main threat, and we test it directly rather than argue around it. If count-based adjustment on a linear probe matched SS-Text+ everywhere, the decomposition would still explain why. But the practical recommendation would collapse to "use the one-line fix," and we would report exactly that. There is reason to expect it fails on missing classes. A class with zero support images has zero count, so the adjustment is undefined without smoothing. The adjustment only shifts scores that the probe never learned for that class, which is the failure Few-Shot, Now for Real attributes to linear probes. The Realistic-regime comparison settles it.

Why not just tune per-class thresholds on a validation split, as in medical coding? Automated Medical Coding on MIMIC-III and MIMIC-IV shows that per-class threshold tuning matters for macro scores, but with thousands of training notes. Few-Shot, Now for Real found that spending scarce images on validation hurt more than it helped. Our validation-split arm measures this for thresholds specifically. The oracle-offset bound tells us the most that thresholds could ever buy, so we can see whether the tax exceeds the gain.

Why not just bootstrap AUCμ instead of deriving a variance? Bootstrapping is valid, and we use it as the reference. The analytic variance is worth deriving because the design multiplies adapters by regimes by datasets by dozens of support draws. Paired, per-comparison bootstraps at that scale are expensive. They are also unstable when a rare class has only a few test instances, which is where the closed form must be checked. If the derivation disagrees with the bootstrap in small classes, we report the bootstrap and treat the derivation as a negative result. The decomposition stands on its own either way.

Why not just report per-class recall tables? Per-class recall conflates the two failures. A rare class can have low recall because its images score poorly, or because every score is shifted toward the common classes. Recall alone cannot tell these apart, and the offset-optimal bound and the logit-oriented ranking metric can.

Decisive Experiments

We use the nine datasets and three regimes of Few-Shot, Now for Real. The histology datasets are NCT-CRC, SICAPv2, and SkinCancer. The ophthalmology datasets are MESSIDOR, MMAC, and FIVES. The radiology datasets are CheXpert, NIH-LT, and the pneumonia data. We use at least 20 support draws per cell and compute macro scores over present classes only. Four checks decide the proposal.

The first check tests the core claim. On Relaxed supports, $\Delta_{\text{bias}}$ for the linear probe should account for most of its gap to SS-Text+. If instead the gap sits in $\mathrm{BA}^\star$ and $\mathrm{AUC}_\mu^{\ell}$, the bias hypothesis is refuted for that regime.

The second check examines Realistic supports. We restrict the decomposition to the missing classes and expect the SS-Text+ advantage to appear as separability, which the validation-free offsets cannot recover.

The third check is the artifact control on NCT-CRC. We fit a color-histogram classifier under the same few-shot supports, following NCT-CRC-HE. We also recompute all separability measures after JPEG and color perturbation. Separability that survives only on clean images is attributed to shortcuts, not morphology.

The fourth check is inference. We recompute every pairwise adapter comparison with the two-level variance and report how many headline differences remain distinguishable from support-draw noise. The analytic AUCμ intervals are validated against bootstrap coverage across class sizes.