Telling Expert Disagreement from Domain Shift in Cross-Modality Segmentation

When a medical segmentation model trained on one imaging modality, say MRI, is run on another, say CT, it becomes uncertain in some regions. That uncertainty has two very different causes. Some regions are hard because human experts themselves disagree there: a fuzzy tumor margin, a thin myocardial wall. Other regions are hard because the model has never seen tissue that looks like this, and its output there is simply unreliable. The first kind should be reported to the clinician as honest ambiguity. The second should be flagged so a human checks it. Current tools blur the two. Multi-rater models learn the experts' disagreement but assume the test images resemble the training images. Domain-adaptation methods do the reverse: they treat every uncertain region as a gap between modalities and push the model to become confident there, which erases legitimate disagreement.

We propose to separate the two with a density test that is told how much the experts disagreed. The test is calibrated on source-modality image regions whose expert disagreement is known. For each target-modality region, we ask whether its feature embedding is unusually sparse compared with source regions that carried the same level of predicted expert disagreement. Ambiguous boundaries are naturally sparse and heterogeneous in feature space, so a global distance threshold flags them as anomalies. A local, disagreement-matched comparison should not. Wrapped in localized conformal p-values, the method can flag "off-support" regions with a controlled false discovery rate and leave genuine ambiguity standing.

Core Claim

The claim is that, under modality shift, expert disagreement and lack of training support cannot be separated by any uncertainty score whose reference distribution ignores disagreement. This holds whether the score is ensemble variance, softmax entropy, or global feature distance.

The reason is that high-disagreement source regions are already the low-density, high-entropy part of the source data. A target region that looks like them is "unusual" in a global sense but normal for an ambiguous region. A disagreement-matched relative density score does separate the two cases, and it comes with finite-sample error control.

This is a deliberately narrow claim. If an ensemble's mutual-information score, calibrated the same way, separates the two cases equally well, the density component is unnecessary. The experiments below are built to find that out.

Positioning Against Prior Work

The motivating evidence comes from two literatures that have not met.

Mind The Gap adapts cardiac and abdominal MRI models to CT. It shows the gap between modalities is spatially uneven, small at the aorta and large at the myocardium. It handles this with an adversarial discriminator mask that pushes alignment hardest in divergent regions, reaching 82.42% Dice on MMWHS MRI→CT. It assumes one ground-truth mask, so the myocardium's divergence and the myocardium's rater ambiguity are treated as the same thing to be minimized.

On the other side, three methods model rater disagreement within a single modality:

  • Pionono gives each rater a Gaussian in latent space. - The Latent Doctor Model conditions a variational prior on the image so its spread tracks panel entropy, with Spearman correlation up to 0.728 on ISUP.

  • PHiSeg uses a hierarchy of latents to produce coherent segmentation hypotheses.

OMBA-UNet and the virtual expert panel show that keeping individual raters beats consensus training. The panel raises F1 from 72.9% to 78.2% and cuts calibration error from 0.183 to 0.050 on Barrett's esophagus. None of these asks what happens to the learned disagreement when the input modality changes. The Latent Doctor Model in particular has no way to tell a borderline case from an unfamiliar one.

The method itself comes from Conditional Testing based on Localized Conformal p-values. That paper weights calibration points by a kernel around the test point, which fixes the "stuck power" of global conformal tests in low-variance regions. Combined with a modified Benjamini–Hochberg procedure, it controls false discovery rate for conditional outlier detection; on health-indicator screening it held error near 0.05 where global thresholding drifted to about 0.09. It has only been tested on tabular and spatial scalar data. We reuse its machinery, but we localize on predicted expert disagreement rather than on raw covariates. That is the key move.

Probabilistic Conformal Distillation is the closest density-and-modality work. It represents missing-modality inputs as Gaussian densities distilled from a full-modality teacher. It needs paired complete-modality data and single labels, and it is a training method rather than a test for unreliable regions. We need neither paired data nor Gaussian latents.

QUBIQ supplies the evaluation setting. Its tasks span MRI and CT with multiple raters, but each task was run in isolation, and it reported that simple deterministic ensembles often beat variational models. That constrains us: an ensemble is the baseline to beat, not a strawman.

Research Program

The pipeline has four stages: a source model, calibration on held-out source data, scoring of target regions, and a decision rule.

Source model. We train a multi-rater model on the source modality. Our primary choice is a deep ensemble with one member per rater's annotations, following QUBIQ's finding. PHiSeg is a second backbone. From the model's encoder we extract one embedding per image patch (or supervoxel), reduced with principal components fit on source data only. This limits the distance concentration that degrades nearest-neighbor densities in high dimension. For each patch the model also predicts a disagreement level $u$: the variance of the predicted rater-wise foreground fractions.

Calibration. On a held-out set of source volumes we compute, for every calibration patch $i$, a relative local density score $s_i$. This is a local-outlier-factor style ratio of a point's k-nearest-neighbor density to that of its neighbors, computed against a separate source reference bank.

Scoring a target patch. For a target patch $x$ with score $s(x)$ and predicted disagreement $u(x)$, we compute a localized conformal p-value:

$$ p(x) = \frac{\sum_{i} w_i \,\mathbf{1}\{s_i \ge s(x)\} + w_0}{\sum_{i} w_i + w_0}, \qquad w_i = K\big(u(x), u_i\big) $$

Here $K$ is a Gaussian kernel on disagreement level with bandwidth $h$, and $w_0 = K(u(x),u(x))$ is the test point's own weight. The effect is that an ambiguous target patch is judged only against source patches of similar ambiguity. Whatever disagreement level the target patch is assigned, the test asks only whether it is unusually sparse for that level. A badly wrong $u(x)$ can therefore make the comparison set itself mismatched, which is why the kernel is also restricted to the anatomical label (below). Patches are numerous, which partly offsets the effective-sample-size shrinkage that localized conformal inference suffers in sparse regions.

Decision rule. Benjamini–Hochberg over all target patches in a volume yields a map of "off-support" regions at a nominal false discovery rate. Inside unflagged regions, the model's predicted rater distribution is reported as is. Inside flagged regions it is withheld and routed to a reader.

Two technical issues need care.

First, patches within a volume are not exchangeable. We split calibration by volume and subsample patches per volume so that weights are not dominated by a few scans. We also report empirical false discovery rate rather than assume the guarantee transfers.

Second, $u(x)$ is itself predicted under shift and may be wrong. We therefore also test a variant that localizes jointly on $u$ and on the anatomical label predicted by the model. We report how often target $u$ falls outside the source range of $u$; when it does, the region is flagged by construction.

Baselines

The naive reference reports the source multi-rater ensemble's predicted distribution everywhere on the target modality, with no flagging. This shows how badly the learned disagreement degrades under shift. It also sets the floor for every downstream metric.

The strongest existing competitor is the deep ensemble's own uncertainty split. Mutual information between members serves as the "model doesn't know" signal and expected entropy as the "experts disagree" signal. QUBIQ found ensembles to be the most faithful reproducers of rater distributions. Mutual information is the standard answer to exactly our question, so this is the baseline most likely to make our method redundant.

To keep the comparison fair, the baseline gets the same ensemble, the same calibration volumes, and the same localized conformal and Benjamini–Hochberg wrapper. Mutual information simply replaces the density score $s$. Hyperparameters ($k$, bandwidth, number of principal components for us; temperature for the ensemble) get the same tuning budget on a source-only validation split, never on target labels. Beating it means a higher area under the ROC curve for detecting off-support regions at matched realized false discovery rate. It also means a lower false-flag rate on regions where the panel genuinely disagrees and the model matches them. Either difference must hold on at least two of the three target settings below.

Global conformal ablation. The same density score with a global (unweighted) conformal p-value isolates the contribution of disagreement-matched localization. The localized conformal paper predicts this version will over-flag ambiguous boundaries and miss subtle shifts in confident regions.

Unconditioned localization ablation. Localizing on feature-space proximity rather than on disagreement isolates whether conditioning on $u$ specifically matters.

Adaptation baseline. Mind The Gap's global-local alignment, applied to the multi-rater source model where feasible, tests whether adapting the features instead of flagging regions preserves or collapses rater variance. On the related single-label task it reports 82.42% Dice on MMWHS MRI→CT, the strongest cross-modality number in the file.

For cross-modality rater-distribution matching on the targets we use, there is no reported result on Lacuna. QUBIQ reports only within-task averages of 0.80–0.85 Dice for top teams. PHiSeg reports a generalized energy distance of 0.2248 on LIDC-IDRI and 0.0864 on prostate MR, all within a single modality.

Why not just…?

Why not just use the ensemble's mutual information as the shift signal? This is the real threat. Our only argument against it is mechanistic. Under shift, ensemble members trained on the same source data can agree confidently on wrong outputs, and their disagreement at ambiguous boundaries is high in both domains. So mutual information should confound the two cases precisely where it matters. The file contains no direct evidence either way for medical modality shift. The matched-wrapper comparison above settles it. If mutual information ties our score, the conclusion is that the contribution is the disagreement-matched conformal wrapper, not the density estimate, and we would report it that way.

Why not just adapt the model to the target modality and skip detection? Adaptation methods such as Mind The Gap and AdaptDiff are trained against a single mask. AdaptDiff reaches 81.94% Dice against an 84.78% oracle on OCTA-500 vision-only transfer. Their objective rewards confident predictions in exactly the divergent, ambiguous regions. We measure this directly by comparing predicted versus empirical rater variance in high-disagreement target regions before and after adaptation. If adaptation preserves that variance and leaves few unreliable regions, flagging is less needed. The two approaches are also complementary, so we additionally run the detector on top of the adapted model.

Why not just threshold a global feature distance such as Mahalanobis or k-nearest-neighbor distance? Ambiguous boundary patches mix tissue classes, so they sit in sparse parts of feature space even in-distribution. A global cutoff will flag them, which is the long-tail failure that density-based outlier work describes. The global-conformal ablation tests exactly this. We predict it shows a markedly higher false-flag rate on high-disagreement, correctly predicted regions at the same detection power.

Proposed Experiments

Primary test: cross-contrast transfer on QUBIQ brain tumor. This task has multiple raters and several co-registered MRI sequences. We assume the per-case annotations apply across sequences, which must be verified. We train on one sequence and test on another. Rater distributions then exist on both sides, so every target region can be labeled.

A region counts as off-support if the generalized energy distance between the predicted and empirical rater distributions exceeds the 95th percentile of that quantity on held-out source data. A region counts as genuinely ambiguous if empirical rater variance is high and the region is not off-support.

We report three outcomes:

  • Detection area under the ROC curve for off-support regions.
  • Realized versus nominal false discovery rate.
  • False-flag rate on genuinely ambiguous regions.

We also report Q-Dice and generalized energy distance on unflagged regions, against the naive reference.

Second test: true MRI→CT shift on MMWHS. Labels here are single masks, so only one half of the question can be checked: whether flagged regions concentrate Dice errors. The myocardium, which Mind The Gap identifies as the most divergent structure, is the stress case.

Third test: injected local shifts. We inject controlled local modality-like artifacts (contrast inversion and noise models of a different physics) into a subset of regions of the QUBIQ prostate MRI and LIDC-IDRI CT test sets. This plants known off-support regions inside genuinely ambiguous ones, so both kinds of error can be counted exactly.

Decisive result. The proposal stands if two things hold on the brain-tumor and injected-shift settings. First, the disagreement-matched score must beat matched-wrapper mutual information on false flags of ambiguous regions without losing detection power. Second, its realized false discovery rate must stay near nominal while the global version's does not.

The proposal fails cleanly if mutual information ties it. It also fails if realized false discovery rate drifts badly because of within-volume dependence. In that case we would report that conformal guarantees do not survive patch-level application to medical volumes, which is itself useful for anyone attempting this.