When a segmentation model trained at one hospital is moved to another, the usual fix is to translate the new images into the look of the old ones and run the old model unchanged. When the old model was trained on several experts' disagreeing outlines, it also reports where experts would disagree on each new image. This proposal argues that, after translation, that reported disagreement mixes two different things. One is real clinical ambiguity, where experts would disagree on this scan. The other is error introduced by the translation step, which sometimes invents or erases small structures. The two call for opposite responses. Real ambiguity should be shown to the clinician and left alone, because more labels will not resolve it. Translator error should be fixed, and a few expert annotations at the new site are the cheapest way to fix it.
The core idea fits in two sentences. First, run several independently trained translators and several learned "rater styles" over each new image; spread across translators is translator error, and spread across rater styles is expert disagreement. Second, spend a scarce expert-annotation budget only on the images and regions where the translator term dominates. We expect this to buy more accuracy per expert minute than spending by total uncertainty, which wastes labels on cases experts themselves cannot settle.
Core Claim
Two lines of work have never been run together. Methods that model disagreement between experts assume all images come from one scanner and site. Pionono embeds each rater as a Gaussian in an 8-dimensional latent space (Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation). Another method pairs a segmentation network with a per-annotator confusion-matrix network (Disentangling Human Error from Ground Truth in Segmentation of Medical Images). The Latent Doctor Model samples expert panels from a conditional variational autoencoder (The Latent Doctor Model for Modeling Inter-Observer Variability).
Conversely, translation-based adaptation treats source labels as clean truth. LE-UDA combines CycleGAN appearance alignment with two teacher networks under 25% source labels (LE-UDA: Label-Efficient Unsupervised Domain Adaptation for Medical Image Segmentation). A CycleGAN is a pair of image generators trained so that a round trip through both returns the original image. LE-UDA's authors note that such translation occasionally hallucinates or deletes small anatomical structures.
Run naively together, the hallucinations show up as "disagreement." The claim is that this confound can be measured and separated without any target-domain labels. The separated translator term can then serve two purposes. It flags hallucinated regions, and it directs a small annotation budget better than total uncertainty does.
Research Program
Train a rater-aware segmenter on a source domain that has multiple expert labels per image. We use a Pionono-style model because it gives each rater a reusable latent vector $z_r$, and we freeze it after training. Following the input-space adaptation recipe of Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors, we then train $K$ unpaired translators $t_1,\dots,t_K$ that map target-domain images into source appearance.
The main family is CycleGANs trained with different seeds and data orderings. A conditional diffusion translator in the style of Translating MRI to PET through Conditional Diffusion Models with Enhanced Pathology Awareness serves as a second family that is stochastic by construction. For a target image $x$, let $p_{k,r}$ be the per-pixel foreground probability predicted for translator $k$ and rater $r$. The law of total variance splits the spread of these predictions into two terms:
$$ \underbrace{\mathrm{Var}_{k,r}[p_{k,r}]}_{\text{total}} \;=\; \underbrace{\mathbb{E}_k\big[\mathrm{Var}_r[p_{k,r}]\big]}_{\text{rater disagreement } U_R} \;+\; \underbrace{\mathrm{Var}_k\big[\mathbb{E}_r[p_{k,r}]\big]}_{\text{translator error } U_T}. $$
Here $U_R$ is how much rater styles disagree on a typical translation. $U_T$ is how much the rater-averaged prediction moves when only the translator changes.
The work has three parts.
The first is validation of the split itself. $U_T$ should rise where the translation is actually wrong, and $U_R$ should track real expert disagreement on the target images.
The second is annotation allocation. Under a fixed expert budget measured in scribbles, the scribbles go to the target images, and the regions inside them, with the highest $U_T$. The segmenter is then fine-tuned on these scribbles with the scribble-only loss of Learning to Segment Medical Images with Scribble-Supervision Alone, which keeps rater latents fixed.
The third is a check that rater latents survive the move. Gap analyses note that Pionono concatenates rater vectors with backbone features that are sensitive to the scanner. We test whether each $z_r$ still reproduces that rater's habits on target images, and whether a $U_T$-guided annotation pass restores them.
The source of disagreement shapes which data we use. LIDC-IDRI provides real disagreement: four radiologists outline lung nodules in thoracic CT (Inter-Rater Uncertainty Quantification in Medical Image Segmentation via Rater-Specific Bayesian Neural Networks). We split it by acquisition scanner and reconstruction setting from image metadata to create a source/target shift; the vendor/kernel effect we are relying on is documented by Inter-vendor harmonization of Computed Tomography (CT) reconstruction kernels using unpaired image translation. For severe, cross-modality shift with known truth, we use BraTS and MM-WHS with synthetic raters whose confusion matrices we control, following the synthetic-rater protocol of the Disentangling paper.
BraTS sequences are co-registered per patient. We withhold the pairing during translator training, but it remains available for evaluation. Translating FLAIR into T1-contrast appearance therefore gives a real T1-contrast image to compare against, which is the ground truth needed to score hallucination maps.
Baselines
The naive reference keeps the frozen rater-aware segmenter and applies it directly to target images. It gets no translation and no target labels. LE-UDA's source-only cardiac baseline reached 17.6% Dice on MM-WHS MRI→CT, which shows how low this floor can be under severe shift.
The strongest adaptation competitor is LE-UDA, run on MM-WHS MRI→CT with 25% source labels. It reports 70.8% Dice there, against 48.3% for SIFA/MT-UDA, and 82.8% on the abdominal CT/MRI benchmark. We rerun it with the same backbone family, data split, and tuning budget, under two label conditions. With clean consensus source labels it serves as a sanity check: our pipeline should land within a few Dice points of 70.8%. Fed STAPLE-fused synthetic-rater labels, it is the real competitor. To beat it, we must match its Dice on the consensus target while also producing a calibrated per-rater distribution, which LE-UDA does not provide.
For target annotation, the strongest competitor is scribble co-segmentation (Scribble-based Domain Adaptation via Co-segmentation). It reports 83.4% Dice on vestibular schwannoma T1-contrast→T2 MRI with scribbles covering about 1% of the volume, above the 81.6% fully supervised target model. We give it exactly the same scribble budget. The comparison is whether $U_T$-guided placement reaches its Dice with fewer scribbles, or exceeds it at equal budget, once synthetic raters with differing margin habits generate the scribbles. We do not expect to beat 83.4% under its original clean-scribble conditions; the test is the noisy-rater setting it never considered.
For the multi-rater side, the reference is the variational inference model of Variational Inference for Quantifying Inter-observer Variability in Segmentation of Anatomical Structures, which reports 0.5372 continuous Dice on QUBIQ brain growth against 0.5083 for Bayes dropout. On LIDC we found no reported number on Lacuna for cross-scanner multi-rater performance. Single-domain OMBA-UNet reports lower generalized energy distance than the Probabilistic U-Net, but the file gives no figures. We therefore establish in-domain Pionono and OMBA-UNet numbers ourselves as the ceiling.
Three ablations isolate the key ingredient, the split. The first allocates by total variance instead of $U_T$. The second allocates by $U_R$, which should be the worst choice. The third uses $K=1$, a single translator, so that $U_T$ is identically zero.
Why not just…?
Why not just fuse the experts' labels with STAPLE and run LE-UDA? This discards the disagreement signal the proposal needs. The Latent Doctor Model shows that majority-vote training makes models overconfident on borderline cases; its distribution-matching model reaches a KL divergence of 0.461 on Gleason 2019. The Disentangling paper reports 14.4% lower confusion-matrix error than STAPLE on BraTS. Fusion is still a fair test of whether the consensus-target Dice of our pipeline is competitive, and it is run as a baseline.
Why not just rank images for annotation by total predictive entropy, or by MC-dropout uncertainty? This is the real threat. If translator error dominates everywhere, total entropy ranks images almost exactly like $U_T$, and the split adds nothing for allocation. The total-variance ablation settles it. We commit in advance that if total-variance allocation comes within one Dice point of $U_T$ allocation across budgets on all three benchmarks, we drop the allocation claim and keep only the hallucination-flagging claim. MC dropout has a second problem: it measures uncertainty in the segmenter's weights, which do not change across translators, so it cannot localize translation error by design. QUBIQ already shows dropout underperforming variational modeling of rater spread (0.5083 vs 0.5372).
Why not just skip generative translation and use intensity normalization or histogram matching? For mild within-CT kernel shift on LIDC this may be enough. If so, $U_T$ will be near zero and the method reduces to Pionono, which is an acceptable and informative outcome. The kernel-harmonization study found vendor effects on emphysema quantification that only generative translation removed ($p<0.05$ before, $p=0.92$ after). Cross-modality shifts such as FLAIR→T1-contrast or MRI→CT cannot be bridged by histogram matching at all. We run histogram matching as an additional naive reference on LIDC to measure this directly.
Proposed Experiments
The first decisive check is whether $U_T$ finds hallucinations. On BraTS, held-out pairing gives a per-pixel error map between each translated image and the real co-registered image. We report the area under the curve for detecting pixels whose translation error exceeds a threshold, comparing $U_T$ against total variance and against $U_R$. Success means $U_T$ clearly beats both; failure means the translator ensemble's spread does not track its error.
The second check is whether $U_R$ tracks real disagreement after the shift. On the LIDC scanner split, we compare $U_R$ on target images against the four radiologists' actual disagreement, scored by generalized energy distance and by rank correlation of per-nodule disagreement. We compare it as well against the uncorrected total variance. The confound hypothesis predicts total variance correlates worse with true disagreement than $U_R$ does, and the gap should widen as the translator gets weaker. We test this by deliberately under-training translators.
The third check is annotation efficiency. On VS, MM-WHS, and BraTS with synthetic raters of differing margin habits, we plot target Dice against the number of scribbles for $U_T$, total-variance, $U_R$, and random allocation, with scribble co-segmentation as the external reference. We also report how well each rater is reproduced on target, as per-rater Dice against that rater's simulated labels, before and after the annotation pass. This tests whether rater latents survive the shift.
The fourth check is cost. We report wall-clock inference for $K$ translators times $R$ raters, given OMBA-UNet's reported 0.34 s per slice against 0.02 s for a U-Net. We check whether $K=3$ to $5$ captures most of the benefit.
References
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Disentangling Human Error from Ground Truth in Segmentation of Medical Images
- The Latent Doctor Model for Modeling Inter-Observer Variability
- LE-UDA: Label-efficient unsupervised domain adaptation for medical image segmentation
- Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors
- Translating MRI to PET through Conditional Diffusion Models with Enhanced Pathology Awareness
- Learning to Segment Medical Images with Scribble-Supervision Alone
- Inter-Rater Uncertainty Quantification in Medical Image Segmentation via Rater-Specific Bayesian Neural Networks
- Inter-vendor harmonization of Computed Tomography (CT) reconstruction kernels using unpaired image translation
- Scribble-Based Domain Adaptation via Co-segmentation
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures