Medical segmentation models that transfer from one imaging modality to another, for example from contrast-enhanced T1 MRI to T2 MRI, or from MRI to CT, are built almost entirely around appearance. The usual recipe first translates image "style" so the labelled source scans look like the unlabelled target scans. It then trains a segmenter and refines it by self-training, meaning it retrains on its own confident predictions (pseudo-labels) for the target scans. That recipe ignores a second change between domains: how much of each scan each structure occupies. Field of view, cropping, slice thickness and patient cohort all differ across modalities and sites. Small or rare structures such as a cochlea, a small tumour or an adrenal gland therefore make up a different fraction of the voxels in target scans than in training. A segmenter's per-voxel probabilities implicitly encode the training fractions. Self-training then feeds those biased predictions back as labels, so the bias compounds.
We propose to estimate, for each target volume, how much of it each structure occupies, without any target labels. The estimate is used to rescale voxel probabilities before pseudo-labels are formed. The key design choice is to estimate the model's confusion behaviour on style-translated source scans, whose labels are known and whose appearance resembles the target, rather than on ordinary source validation data. We also combine each noisy per-volume estimate with the cohort average, so that a single scan never drives an extreme correction. The core test is simple. If target ground-truth structure fractions, plugged in as an oracle, do not improve small-structure Dice on CrossMoDA, MM-WHS and AMOS, the idea is wrong and we stop.
Core Claim
Label distribution shift (also called prior shift) is the setting where class frequencies $P(Y)$ change between training and deployment while the appearance of each class, $P(X\mid Y)$, stays fixed. Under this shift a trained classifier over-predicts classes that were common in training. Our claim is that in unsupervised cross-modality 3D segmentation, a measurable part of the remaining error on small structures is prior shift of this kind, not appearance shift. Correcting it inside the self-training loop yields gains that style translation, size-aware pseudo-label filtering and Dice-based training do not already deliver.
The correction is the standard Bayesian prior swap. For voxel $i$ in target volume $v$ with source-model posterior $p(y\mid x_i)$, the rectified posterior is
$$ \tilde p(y\mid x_i) \propto p(y\mid x_i)\,\frac{\hat\pi_v(y)}{\pi_S(y)}, $$
where $\hat\pi_v$ is the estimated fraction of voxels of class $y$ in volume $v$ and $\pi_S$ is the effective training prior. The novelty lies in how $\hat\pi_v$ and $\pi_S$ are obtained in the dense 3D regime, where the classical tools break down.
The first detail concerns the training prior. Patch-based trainers such as nnU-Net oversample foreground patches, so $\pi_S$ is not the dataset's voxel fraction. It is the class frequency seen by the network under its patch sampler, and we measure it directly from the sampler.
The second detail concerns the target prior. We estimate $\hat\pi_v$ by solving
$$ \hat\pi_v = \arg\min_{\pi \in \Delta_K} \lVert C_{\mathrm{tr}}\,\pi - q_v \rVert_2^2 + \lambda\,\mathrm{KL}(\pi \,\Vert\, \bar\pi). $$
Here $q_v$ is the mean soft prediction over voxels of $v$, $C_{\mathrm{tr}}$ is the soft confusion matrix measured on translated source volumes, $\Delta_K$ is the probability simplex over $K$ classes, $\bar\pi$ is the running cohort-level estimate, and $\lambda$ sets how strongly each volume is pulled toward that cohort estimate. Restricting the solution to the simplex removes the negative weights that plain matrix inversion produces under sampling noise. The pull toward the cohort estimate handles the near-singular confusion matrices that arise when background outnumbers a structure 99 to 1, and it gives a usable estimate from the first scan rather than after a long warm-up.
Relation to Prior Work
Black Box Shift Estimation (Detecting and Correcting for Label Shift with Black Box Predictors) estimates target priors by inverting a source-validation confusion matrix. It supplies our estimator's skeleton. It also carries its limits: it assumes independent class draws, an invertible matrix and an unchanged $P(X\mid Y)$, and all three fail at the voxel level across modalities. Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift proves that aligning feature distributions between domains with different class priors leaves an irreducible error. It relaxes invariance to a learned representation, $P_S(Z\mid Y)=P_T(Z\mid Y)$. This motivates the proposal. A translation network such as CycleGAN or Contrastive Unpaired Translation is exactly such a distribution-matching step, and that work's relaxed assumption is our reason to measure the confusion matrix on translated rather than raw source data.
The closest work is Undoing the Damage of Label Shift for Cross-domain Semantic Segmentation. It applies the same prior-ratio rescaling to 2D driving scenes, using one target prior pooled offline over the whole target set, and raises FADA from 46.9% to 48.8% mIoU on GTA5 to Cityscapes. We differ in four ways. We estimate a prior per 3D volume with cohort shrinkage, not once per dataset. We use a confusion-corrected estimate rather than pooled raw predictions. We use the patch-sampler prior for $\pi_S$. And we place the correction inside pseudo-label generation for medical cross-modality pipelines, which that work never tested.
On the medical side, Test-time Adaptation with Calibration of Medical Image Classification Nets for Label Distribution Shift handles prevalence shift for whole-scan CT classification with an expert committee reweighted by augmentation consistency. It shows that medical label shift is real, but it covers neither dense outputs nor modality change. ODS and Label Shift Adapter separate feature adaptation from prior correction at test time, but only on 2D natural-image classification. They constrain our design: the Label Shift Adapter documents that entropy minimisation under prior shift becomes "confidently wrong", which is why we correct priors rather than adapt by entropy.
The cross-modality pipelines are our application regime. These are the CrossMoDA winners (2021; CrossMoDA 2021 challenge: Benchmark of cross-modality domain adaptation techniques for vestibular schwannoma and cochlea segmentation, 2021–2023), MS-MT, FPL+ and Meta-hallucinator. All of them treat translation plus pseudo-label filtering as sufficient. FPL+ is the most relevant contrast because its "size-aware" uncertainty filter is an implicit, hand-tuned response to the same small-structure frequency problem we model explicitly.
Research Program
The work has three stages, and each stage gates the next.
The first stage is a diagnosis. On benchmarks where target labels exist, we measure how much the true per-volume structure fractions in the target differ from the effective training prior $\pi_S$. We then rescale the outputs of a fixed source pipeline (translation, then nnU-Net) using the oracle target fractions. If the oracle gain on small structures is negligible, prior shift is not the bottleneck and the program ends there, which is itself a publishable negative result about the roughly 88% Dice plateau.
The second stage builds the estimator described above. We compare confusion matrices measured on translated source volumes against matrices from source validation data. We vary the pull toward the cohort prior from zero to strong, and we measure estimation error against the oracle fractions as a function of how many target volumes have been seen.
The third stage inserts the rectified posteriors into self-training. Pseudo-labels for each round are taken from $\tilde p$ instead of $p$. We test whether this stops the drift in which small structures shrink over successive rounds, which we will measure directly as the predicted-volume trajectory across rounds.
Baselines
The naive reference is the source pipeline without self-training: translation followed by an nnU-Net segmenter applied directly to target scans. A second reference adds standard self-training with confidence-thresholded pseudo-labels. These show how much self-training helps and whether it erodes small structures.
The strongest existing competitor is FPL+ (FPL+). It is the most complete published cross-modality method in the file, combining translation, Dual Batch Normalization (separate normalisation statistics per modality with shared filters), uncertainty-filtered pseudo-labels and size-aware filtering. It reports 91.98% Dice on hrT2 to ceT1 adaptation, against 84.29% for DAR-Net and 67.48% for SIFA. We will rerun FPL+ from its description on identical splits, with the same nnU-Net backbone, the same number of self-training rounds and the same hyperparameter search budget as our method. We will then add our rectification on top of FPL+, so the comparison isolates the prior correction. Beating it means a statistically reliable gain in small-structure Dice (cochlea; adrenal glands and esophagus on AMOS; pulmonary artery on MM-WHS) across five seeds, without loss on large structures.
For CrossMoDA ceT1 to hrT2 we also report the published references. Top unsupervised median Dice was 87.0% for VS and 85.7% for cochlea in CrossMoDA 2021, against a supervised upper bound of 92.5% and 87.7%. MS-MT reports mean Dice of 83.8% for VS and 81.4% for cochlea on the 2022 validation set. For MM-WHS unsupervised adaptation, no reported FPL+ numeric result was found on Lacuna. Meta-hallucinator reports only relative gains in the few-shot setting.
The prior-correction competitor is Inference Adjustment from Undoing the Damage of Label Shift: one dataset-level target prior obtained by smooth max pooling, applied in the same pseudo-labelling slot as ours.
Four ablations isolate each ingredient. The first takes the confusion matrix from source validation instead of translated source data. The second removes the pull toward the cohort prior, which reduces to per-volume BBSE on the simplex. The third uses the dataset voxel prior instead of the patch-sampler prior for $\pi_S$. The fourth applies rectification at inference only rather than inside self-training. The oracle-prior run serves as the upper bound.
Why not just…?
Why not just use Dice loss, which already ignores class frequency? This is the most serious threat. Dice-trained networks are less tied to voxel priors than cross-entropy networks, and nnU-Net combines both losses. If our oracle diagnosis shows little gain for Dice-plus-cross-entropy nnU-Net, the threat wins. We will therefore run the diagnosis separately for cross-entropy-only, Dice-only and combined training. We report where the remaining prior sensitivity lives, and we will drop the claim for any setting where the oracle gives no gain.
Why not just rely on FPL+'s size-aware filtering? Size-aware filtering decides which pseudo-labels to keep, but it does not move the decision boundary. If the model already under-segments a small cochlea, filtering keeps an under-segmented label. The "FPL+ plus rectification" run settles this directly. A null result there would mean filtering already absorbs the prior shift.
Why not just pool one target prior over the whole dataset, as Undoing the Damage does? Structure fractions vary strongly between patients, for example in tumour size. A single pooled prior over-corrects some volumes and under-corrects others, and it needs the full target set offline. The per-volume versus dataset-level comparison, stratified by tumour volume, tests whether the per-volume estimate is worth its extra noise.
Why not just use TENT-style entropy minimisation? On VisDA-C with skewed class frequencies, TENT reaches 48.68% accuracy against 72.97% for the prior-aware Label Shift Adapter (Label Shift Adapter). We include TENT on the segmenter as a baseline to check whether that failure carries over to dense prediction.
Proposed Experiments
We use three benchmarks. CrossMoDA (ceT1 to hrT2, scored on the challenge validation server, plus the reverse hrT2 to ceT1 direction used by FPL+ where labels exist) covers a tiny structure and a tumour of variable size. MM-WHS 2017 (MRI to CT, labels in both modalities) covers seven cardiac substructures. AMOS (CT to MRI, 15 organs, where the benchmark reports drops above 20% on unseen MRI scanners in AMOS) stresses the multi-class confusion matrix with many rare organs.
To create controlled prior shift without changing appearance, we make three kinds of target splits. On MM-WHS and AMOS we vary the cropping and field of view of target volumes. On labelled data we resample target cohorts by structure volume. On CrossMoDA we stratify by small and large tumours. We also keep a no-added-shift condition, to check the cost of the correction when there is nothing to correct; ODS reports a penalty in that regime.
Metrics are per-structure Dice and average symmetric surface distance, error of the prior estimate against true fractions, predicted-volume drift across self-training rounds, and expected calibration error.
The decisive checks are three. The oracle-prior gain must be substantial on small structures. The estimated-prior gain must recover a large share of that oracle gain. And the gain must survive on top of FPL+ under Dice-plus-cross-entropy training. Failing the first ends the program. Passing the first but failing the third would show that existing filtering already absorbs the shift.
References
- Detecting and Correcting for Label Shift with Black Box Predictors
- Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift
- Undoing the Damage of Label Shift for Cross-domain Semantic Segmentation
- Test-Time Adaptation with Calibration of Medical Image Classification Nets for Label Distribution Shift
- ODS: Test-Time Adaptation in the Presence of Open-World Data Shift
- Label Shift Adapter for Test-Time Adaptation under Covariate and Label Shifts
- CrossMoDA 2021 challenge: Benchmark of cross-modality domain adaptation techniques for vestibular schwannoma and cochlea segmentation
- crossMoDA Challenge: Evolution of Cross-Modality Domain Adaptation Techniques for Vestibular Schwannoma and Cochlea Segmentation from 2021 to 2023
- MS-MT: Multi-Scale Mean Teacher with Contrastive Unpaired Translation for Cross-Modality Vestibular Schwannoma and Cochlea Segmentation
- FPL+: Filtered Pseudo Label-Based Unsupervised Cross-Modality Adaptation for 3D Medical Image Segmentation
- Meta-hallucinator: Towards Few-Shot Cross-Modality Cardiac Image Segmentation
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation