A model trained on one kind of medical image is often reused on another: a network built for chest X-rays gets pointed at skin photographs or retinal scans. The new domain has its own rare diseases, and experts have labeled only a handful of examples of each. In-domain long-tail studies found something useful. When a network is trained on naturally imbalanced data, its internal features stay good, and the rare-disease failures come almost entirely from the final classification layer. Retraining just that layer with class-balanced sampling fixes much of the damage. This proposal asks whether that finding survives a change of imaging modality. If it does not, and rare-class failure moves into the features themselves when the modality changes, then the cheap fix stops working. What is needed instead is an encoder trained, across many source modalities, specifically so that a balanced last layer fitted from a few imbalanced expert labels still separates the rare classes of a modality it has never seen.
The work has two parts. First, a diagnostic that locates tail failure (classifier vs. features) under a leave-one-modality-out protocol built from existing long-tailed medical benchmarks. Second, a targeted method that is only worth building if the diagnostic says the features are the problem. That method is episodic meta-training on deliberately imbalanced tasks, with a class-balanced head solved inside each episode. The diagnostic can kill the method. That is intended.
Core Claim
Our hypothesis is that under cross-modality transfer with scarce, imbalanced target labels, most of the tail-class error sits in the representation rather than in the classifier. We therefore claim that an encoder meta-trained to support class-balanced few-label heads will recover more target-modality tail accuracy per expert label than any recipe that re-balances only the classifier.
The in-domain evidence for the opposite view is strong. Holste et al. benchmark long-tailed chest X-ray classification and show that classifier weight norms scale with class counts. Their classifier retraining (cRT) lifts tail accuracy on NIH-CXR-LT from 1.7% to 30.0%. That paper supplies both the method we test against and the null hypothesis. Its study stays inside one modality, with thousands of training images per head class.
MONICA benchmarks long-tailed methods across 12 datasets in six medical domains and finds they also help out-of-distribution detection. MONICA motivates the setting but never transfers across domains, and it assumes fully labeled training sets. Neither paper tells us what happens when the encoder has never seen the target modality and the target tail has one to five labels.
Research Program
Stage 1: build the protocol. We use the MONICA datasets as a pool of imaging modalities. These are radiography (CheXpert, NIH-CXR-LT, OrganAMNIST), pathology and histology (PathMNIST, TissueMNIST), hematology (BloodMNIST), dermatology (ISIC-2019-LT) and ophthalmology (ODIR, RFMiD, with Eyepacs/Messidor-2 as a further check). In each rotation, one modality family is held out entirely.
On the held-out modality, the expert budget is a per-class cap $k$ drawn from {1, 5, 20}. Head classes receive $k$ labels, and tail classes receive $\min(k, n_c)$, where $n_c$ is the class's true count. This keeps the target support set imbalanced in the way clinical label collection is. All inputs are resized to a common resolution. Evaluation uses each dataset's full held-out test split and reports:
- head, medium and tail accuracy, using Holste's frequency bins;
- the group-wise average of those three;
- worst-class accuracy;
- a labels-to-target curve: how many expert labels per class are needed to reach a fixed tail accuracy.
Stage 2: the diagnostic. All methods start from one encoder, pretrained with ordinary training on the pooled source modalities. On the held-out modality we compare three things:
- Few-label cRT: a class-balanced linear head fit on the $k$-capped labels.
- Oracle probe: a class-balanced linear head fit on every available target label, with the encoder still frozen.
- In-domain reference: the same two probes on an encoder that did see the target modality in pretraining.
The gap between few-label cRT and the oracle probe measures what better head-fitting could ever recover. The gap between the oracle probe and the in-domain reference measures the representation deficit caused by the modality shift. If the second gap is small for tail classes, Holste's decoupling holds across modalities, and Stage 3 is unnecessary. In that case the right paper is a negative result plus a label-budget study of cRT.
Stage 3: meta-training on imbalanced episodes. This stage runs only if the diagnostic shows a representation deficit.
- Sample an episode. Each episode is a task from one source modality. It uses a random subset of that modality's classes, and its support counts follow a power law so that it looks like a clinical long-tailed label set rather than the balanced N-way K-shot episodes meta-learning usually uses.
-
Fit the head inside the episode. We solve a class-balanced ridge head in closed form on encoder features $f_\theta(x)$, where $\theta$ are the encoder weights: $$W^*(S)=\arg\min_W \sum_{c}\frac{1}{n_c}\sum_{i\in S_c}\lVert W f_\theta(x_i)-y_i\rVert^2+\lambda\lVert W\rVert^2 .$$ Here $S_c$ is the support set for class $c$, $n_c=|S_c|$, $y_i$ is the one-hot label, and $\lambda$ is a ridge weight. The $1/n_c$ weighting is cRT's balancing written as a loss. Because the solution is closed-form, it is differentiable in $\theta$.
-
Update the encoder. The encoder minimizes the class-averaged query loss: $$\min_\theta\ \mathbb{E}_{T}\Big[\tfrac{1}{C}\sum_{c=1}^{C} L_c^{\text{query}}\big(W^*(S),\theta\big)\Big].$$ Here $C$ is the number of classes in task $T$, and $L_c^{\text{query}}$ is the query loss on class $c$. Tail classes therefore count as much as head classes.
The method borrows three things from prior work, and changes each:
- Episodic outer loop. This comes from MAML, which meta-learns an initialization adapted by inner gradient steps. We replace those gradient steps with a balanced closed-form head, because unbalanced inner-loop gradients are exactly what inflates head-class weight norms.
- Worst-case variant. The worst-case objective comes from Robust Neural Processes, which optimizes the conditional value-at-risk (the average over the worst fraction) of task losses. We test a variant where the outer average over classes is replaced by the average over the worst fraction of classes. This is a cheap per-class version that avoids their double-loop optimizer.
- Contrast with the closest medical meta-learner. SSM-SAM meta-learns adapters for few-shot CT/MRI segmentation with MAML++, but with balanced support sets and dense masks. We keep the task imbalanced and make the class-balanced adaptation rule the thing the encoder is trained for.
We deliberately avoid pixel-space mixing. Holste et al. report that MixUp hurts on radiographs because blending erases thin pathological edges, even though Balanced-MixUp helps on fundus images.
Baselines
Naive references. There are two. The first is the source-pretrained encoder with an ordinary head trained by plain cross-entropy on the $k$-capped target labels. This is the direct analogue of the Softmax baseline that reaches 1.7% tail accuracy in-domain on NIH-CXR-LT in Holste et al.. The second is the same setup with full end-to-end fine-tuning.
Few-label cRT and τ-normalization. These are the decoupling fixes from Holste et al., applied on the frozen source encoder. They are the strongest competitor on the precise question, because they embody the claim we are testing. Their reported in-domain result is 30.0% tail accuracy on NIH-CXR-LT and 36.7% on MIMIC-CXR-LT. We found no reported result on Lacuna for cRT under leave-one-modality-out transfer.
GCL. This is the long-tailed method with the strongest number in the file: 71.33% tail accuracy on ISIC-2019-LT versus 12.67% for ordinary training, in MONICA. We run it in two ways: as the source-pretraining objective, and as the target adaptation objective.
Target-adapted two-stage. Fine-tune the whole encoder on the target's capped labels with a balanced loss, then apply cRT. This is the practitioner's default.
Balanced-MixUp. The method of Galdran et al., which reaches 80.78 quadratic weighted kappa on Eyepacs versus 74.45 for class-balanced sampling. We use it during target adaptation.
Fairness controls. Every method uses the same source images, the same architecture (ResNet-50), the same pretraining compute measured in GPU-hours, the same target labels, and the same hyperparameter search budget. That budget is 20 trials, tuned on a validation split drawn from the source modalities only, never the held-out one.
What counts as beating a baseline. The proposal beats the baselines if, averaged over held-out rotations, it improves tail accuracy over the best of cRT, GCL and target-adapted two-stage at $k \le 5$. The improvement must exceed the across-seed spread, and head accuracy must not drop by more than a point.
Ablations. Two ablations isolate the key ingredient. The first uses balanced episodes with the same balanced inner head, testing whether imbalanced episodes matter. The second uses imbalanced episodes with an unweighted inner head, testing whether in-loop balancing matters. The worst-class outer loss is a third, optional variant.
Why not just…?
Why not just retrain the classifier with cRT on the target modality? This is the hypothesis that could make the whole method unnecessary, which is why Stage 2 tests it first. If the oracle probe on frozen source features nearly matches the in-domain reference on tail classes, cRT is the answer and we report that. The method is justified only where that gap is large.
Why not just fine-tune the encoder on target data with a balanced loss, then apply cRT? This is a real threat whenever head classes are well labeled, because head data can adapt the features to the new modality. The concern is the tail. With one to five labels, re-balancing overfits to incidental background detail, which is the failure Galdran et al. document for class-balanced sampling (74.45 vs 80.17 kappa). We run this baseline at every budget. We also run it on top of our meta-trained encoder, since the two methods can combine. If the gain vanishes at $k=20$ but holds at $k \le 5$, the proposal's value is specifically in the scarce-label regime.
Why not just use Balanced-MixUp or oversampling? Both act on the data balance, not on whether the features for an unseen modality separate rare classes. Pixel mixing is also known to hurt on radiographs (Holste et al.). We include both methods on each held-out modality, so fundus rotations, where MixUp is known to help, give it its best chance.
Why not just use a broad medical foundation model? Med-MLLM reports deployment with 1% labeled data across imaging types. Such models, however, may have seen the "held-out" modality, which confounds the question. Wolf et al. also show that medical vision-language models lean on memorized priors rather than image evidence. We report a frozen foundation-encoder arm with cRT as a practical reference, but keep it out of the controlled comparison.
Why not just use standard balanced-episode meta-learning? Balanced N-way K-shot episodes never expose the encoder to the imbalance it will face at deployment. The first ablation above is this alternative exactly.
Decisive Checks
Three outcomes settle the proposal.
The diagnostic itself. If the representation deficit on tail classes is small across rotations, Holste's decoupling extends across modalities. That is a publishable result, and we stop.
Whether the method helps. If the deficit is large, the meta-trained encoder must shrink it on the held-out modality without seeing that modality, and do so mostly on tail classes. If gains are uniform across head and tail, the effect is generic transfer, not what we claim.
Which ingredient matters. Both ablations must lose to the full method. Otherwise the claimed mechanism is wrong even if the numbers improve.
The labels-to-target curve turns all three results into the quantity a clinical team cares about: how many expert annotations of a rare finding a new imaging modality actually requires.
References
- Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study
- MONICA: Benchmarking on Long-tailed Medical Image Classification
- Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
- Learning Robust Neural Processes with Risk-Averse Stochastic Optimization
- Self-Sampling Meta SAM: Enhancing Few-Shot Medical Image Segmentation With Meta-Learning
- Balanced-MixUp for Highly Imbalanced Medical Image Classification
- A medical multimodal large language model for future pandemics
- Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images