Who Gets Labeled Decides Who Gets Segmented: Support-Set Composition in Few-Shot Cross-Modality Segmentation

Some medical segmentation methods now learn to outline heart chambers in CT scans after seeing labels on only one to four MRI scans. They do this by translating the few labeled scans into the look of the other scanner type and training on the result. Averaged over test patients, these methods work surprisingly well. What they cannot do is invent anatomy that was never shown to them. When a model's only labeled examples come from four patients, those four people's hearts become its template for every heart, and patients whose anatomy differs (by body size, sex, or age-related remodeling) may be systematically outlined badly. Nobody has measured this. The published benchmarks draw one random set of labeled scans and report the average score.

This note argues that in this regime, which scans get labeled matters for subgroup fairness at least as much as which adaptation algorithm is used, and that it can be controlled cheaply. We propose two things. The first is an audit protocol that measures how a target patient's error depends on their anatomical distance from the labeled examples. The second is a label-free rule for picking which few scans to annotate: it spreads the annotation budget across the range of anatomy seen in the unlabeled scans of both modalities, with no demographic metadata required.

Core Claim

The claim has two parts. First, in 1–4 shot cross-modality segmentation, per-patient error on the target modality rises with that patient's morphological distance to the nearest labeled source scan. This happens because the translated training images carry the donor's anatomy and only change its appearance. Across random draws of the labeled set, the error on the worst-served patients should vary much more than the mean. Second, choosing the few labeled scans to cover the morphology of the unlabeled pool reduces worst-group error and its variance across draws, without lowering mean Dice. The effect should hold whether the adaptation method is a translation-plus-teacher pipeline, a meta-learned hallucinator, or a graph-based few-shot segmenter. If this holds, the fix sits upstream of every existing method and costs nothing at training time.

What Prior Work Does and Leaves Open

The methods we audit come from the few-shot cross-modality line. MT-UDA combines CycleGAN-style image translation (a network that converts images from one scanner's appearance to another's) with two "teacher" networks that keep predictions consistent, trained from 4 labeled MR scans on MM-WHS 2017. Meta-hallucinator meta-learns a module that generates synthetic intermediate training images and adds consistency constraints, working from 1–4 source scans. C-Graph represents organs as graphs of connected regions and matches those graphs from CT to MRI in the few-shot setting.

All three report aggregate Dice and surface distance on a fixed support set, meaning the small set of labeled examples the model learns from. None reports anything about the patients in that set or in the test set. We reuse all three unchanged as the adaptation methods. Our contribution is the support-selection layer and the evaluation around it.

Two findings motivate the hypothesis. The CheXzero bias study shows that high-average-accuracy medical models still underdiagnose female, young, and Black patients, and most severely at their intersections, where data are thinnest. Unpaired 3D facial attribute translation with a geometric map shows that a generator trained on one population produces localized geometric artifacts on another. That is the translation-carries-donor-anatomy failure we expect, seen in a different domain.

Two further papers constrain the design. The small-data COVID chest X-ray study found that models cluster by acquisition site rather than pathology. Its authors explicitly lacked demographic metadata to separate demographic from acquisition bias. This is why our method must not depend on demographic labels, and why the audit must separate scanner effects from anatomy. The CHAOS challenge documents a drop from about 0.97 to 0.88 Dice for joint CT/MR models. It also shows surface-distance errors around 20–30 mm even when Dice exceeds 0.95, so we report surface distance and not only overlap.

The closest fairness-under-limited-adaptation work is the CNN-XGBoost adapter for chest X-ray, which halves an Equalized Odds gap by retraining a small head. That work is single-modality, uses large cohorts, and needs group labels. Our setting has none of these.

Research Program

The central object is a morphology descriptor computable on unlabeled scans of either modality. For cardiac data, the descriptor includes thoracic cross-sectional extent, field-of-view-normalized heart bounding-box dimensions from a coarse detector, and a threshold-based blood-pool volume estimate. All are standardized within modality so that scanner contrast does not drive them. As an alternative descriptor, we use features from an encoder pretrained on mixed CT and MRI, following the autoregressive pre-training approach. The comparison between the two descriptors is itself informative: appearance-heavy features may cover scanners rather than anatomy.

The audit has four steps. For each adaptation method, we draw many random K-scan labeled sets (K = 1, 2, 4), run the full pipeline, and record per-patient Dice and average surface distance (ASD) on the target modality. We then regress per-patient error on the descriptor distance to the nearest labeled scan, controlling for global image-quality covariates. Separately, we run a controlled shift: all labeled scans come from the lowest tercile of the descriptor, and we test on the highest tercile, and vice versa. Finally, we inspect translated images directly, measuring whether the chamber volumes in synthetic target images track the donor's labels (expected) or the target population's range (which would indicate the translator adds anatomical diversity).

The selection rule is k-medoids with k = K. It runs on descriptors from the pooled unlabeled source and target scans, and each medoid is mapped to its nearest source scan, which is the one sent for annotation. The rule optimizes coverage of the target population's anatomy using only unlabeled data.

Where a dataset release includes age or sex, we use them only for evaluation, never for selection. Where it does not, subgroup results are stated in morphological terms, and we do not claim they are demographic findings.

Baselines

The naive reference point is random selection of the labeled scans. This is what the benchmarks already do. To make it fair, we run it over at least 30 draws per K and report the full distribution rather than one draw. The strongest existing adaptation method on the cardiac benchmark is Meta-hallucinator. The file reports only its improvement: 6.7% higher Dice and 0.9 mm lower ASD than MT-UDA in the 4-shot MR→CT setting. MT-UDA itself reports Dice 61.6% and ASD 5.9 mm there, against 42.7% and 11.3 mm for SIFA (MT-UDA).

For abdominal CT→MRI, C-Graph reports 72.83% Dice (C-Graph). We rerun these methods with their published settings and identical tuning budgets for every selection rule. Only the identity of the labeled scans changes, so any difference is attributable to selection.

The metadata-stratified baseline picks one labeled scan per age or sex stratum wherever those labels exist. It is the obvious upper reference for group-aware selection.

The key ablation replaces the morphology descriptor with pretrained appearance features, and separately with uniform-random selection from the source pool alone, which ignores the target pool. This isolates whether the gain comes from covering anatomy, covering scanners, or covering the target population specifically.

Beating the baselines means the following. Worst-tercile Dice improves by more than the standard deviation of worst-tercile Dice across random draws. The slope of error against distance-to-support flattens. Mean Dice does not fall below the random-draw median. All of this must hold for at least two of the three adaptation methods.

Why not just…?

Why not just use heavy scale and elastic augmentation? Random resizing and deformation of the few labeled hearts might cover size variation without any selection. This is a real threat for global size differences. Augmentation seems less likely to reproduce correlated shape changes such as chamber proportions and wall thickness. The COVID small-data study found that standard preprocessing did not prevent shortcut learning, although that was a contrast-normalization rather than a shape-augmentation result. We therefore run random selection with aggressive shape augmentation as a direct competitor. If it matches morphology-covering selection on the tercile-shift test, we will report that augmentation suffices.

Why not just stratify the labeled set by recorded age and sex? Where such labels exist, this is simple and may work as well. Our selection rule is designed for the common case where they are missing or coarse, a limitation the COVID study and the CheXzero study's intersectional findings both point to. We include stratified selection as a baseline. If it matches our rule, the practical conclusion is still useful: morphology descriptors are an adequate stand-in when metadata are missing.

Why not just use a foundation-model adapter such as SSM-SAM? Self-Sampling Meta SAM freezes a large pretrained segmenter and meta-learns small adapters, gaining 10.21% Dice on ABD-30. However, it too adapts from a handful of labeled examples, so the same support-composition question applies to it. The CheXzero study also shows that large pretrained models can carry demographic disparities rather than remove them. We treat an adapter-based model as a fourth adaptation method under the audit, not as an alternative to it.

Why not reweight or mix across groups during training? Distance-aware mixup reduced error disparity from 0.48 to 0.01 on text (DMIX). With four labeled volumes, however, there are no groups to reweight or mix. Selection is the only lever available before training.

Proposed Experiments and Decisive Checks

The primary benchmark is MM-WHS 2017, MR→CT and CT→MR, with K = 1, 2, 4. The secondary benchmark is abdominal CT→MRI in the CHAOS and C-Graph settings. Because target test sets are small, a limitation the file flags, we avoid binning into small groups. The main statistic is the per-patient regression slope of Dice and ASD against distance-to-support, with patient-level bootstrap confidence intervals and draw-level variance reported alongside.

The hypothesis is falsified if the tercile-shift experiment produces no drop beyond draw-to-draw noise, since then support composition does not matter. It is also falsified if the slope is flat under random selection, since then there is no donor-template effect to fix. Finally, it is falsified if augmentation with random supports closes the worst-tercile gap as well as covering selection does.

The single most informative check is whether the chamber volumes in translated synthetic images stay locked to donor labels. If they do, every translation-based few-shot method inherits its labeled patients' anatomy by construction, and selection becomes the natural remedy.