When a segmentation model trained at one hospital is run on scans from another, it often makes confident mistakes. It also tends to be confident on the parts of an image where human experts genuinely disagree, such as fuzzy tumor margins or faint vessel ends. These are two different kinds of uncertainty. The first comes from the scanner and protocol having changed. The second comes from the anatomy itself being ambiguous. Current tools blur them together, and on 3D scans even estimating total uncertainty is expensive, because standard approaches need several full-volume forward passes. This note proposes a small add-on that sits on top of a frozen 3D segmenter. It predicts, voxel by voxel, how much a panel of experts would disagree. That prediction serves two purposes. The model's confidence is rescaled to match it, and any uncertainty left over beyond it is reported as a warning that the scanner has changed.
The main design choice is that this "expert disagreement predictor" reads mostly the shape of the predicted mask and its local boundary context, and only a coarse, intensity-normalized view of the image. Mask geometry changes far less across scanners than raw MRI intensities do. The predictor is trained on multi-rater data, meaning scans annotated independently by several experts. It works on thin stacks of neighboring slices whose physical spacing is given as an input, so it costs a fraction of one 3D forward pass and does not assume a fixed slice thickness.
Core Claim
The closest existing method is Improved post-hoc probability calibration for out-of-domain MRI segmentation. It fixes overconfidence without retraining by predicting a per-pixel temperature map, which is a voxelwise rescaling of the model's confidence. It builds this map from two cues: sensitivity to brightness and contrast jitter, and a denoising autoencoder trained on clean masks that flags implausible shapes. The research file notes that this shape prior needs regular geometry such as cardiac chambers and cannot regularize tumors or vascular trees. We keep the post-hoc, frozen-backbone setting. We replace the global template with a learned, local model of where experts disagree, and we change the calibration target from a hard consensus mask to the experts' soft vote.
Our claim is that calibrating to a disagreement map predicted from mask geometry does two things. First, it transfers across scanners better than calibration based on intensity perturbations or a shape template. Second, it separates human ambiguity from domain-shift uncertainty well enough that the leftover term flags failing cases.
Concretely, let $z_i$ be the frozen model's foreground logit at voxel $i$. Let $a_i$ be the fraction of raters who labeled voxel $i$ foreground. Let $\hat a_i$ be the disagreement predictor's estimate of $a_i$. The calibrator outputs a positive temperature $T_i$ and is trained on source data under simulated acquisition shifts to minimize
$$\mathcal{L} = \sum_i \big(\sigma(z_i/T_i) - a_i\big)^2,$$
where $\sigma$ is the logistic function. This is a Brier score (squared error between predicted probability and outcome) against the rater soft label rather than against the consensus mask. At test time we report an excess-uncertainty score for each voxel:
$$e_i = H\big(\sigma(z_i/T_i)\big) - H(\hat a_i).$$
Here $H$ is binary entropy, a measure of how uncertain a probability is. A positive $e_i$ means the model is more uncertain than expert disagreement alone would explain. We aggregate $e_i$ over each volume to flag cases likely damaged by domain shift.
Research Program
The work has three parts.
The first part is a measurement study that is currently missing. Multi-rater segmentation models each learn rater behavior in a different way. Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation gives each rater a Gaussian in an 8-dimensional latent space. Diversified and Personalized Multi-Rater Medical Image Segmentation learns a consensus core and an outer envelope. The Latent Doctor Model for Modeling Inter-Observer Variability models rater style with a conditional variational autoencoder. Multi-rater Prompting for Ambiguous Medical Image Segmentation adds a learned prompt per rater. All four are evaluated in-distribution. We ask what happens to their predicted disagreement under scanner shift: does it shrink, grow, or move away from the boundaries where experts actually disagree? This study motivates the proposal. If these models' spread already tracks true disagreement across scanners, the method below is unnecessary.
The second part is the disagreement predictor and calibrator. The predictor takes three inputs: the frozen model's predicted mask, its signed distance transform, and a heavily normalized, low-resolution copy of the image. It processes 3 to 7 neighboring slices at a time, following the 2.5D slab idea the file attributes to Shi et al. The physical slice spacing is passed as an explicit input. This addresses a known weakness of fixed-distance inter-slice matching such as the contextual embedding block in Contextual Embedding Learning to Enhance 2D Networks for Volumetric Image Segmentation, which degrades when slice thickness changes. The calibrator is a shallow network in the style of Ouyang et al. It takes the frozen logits, the predicted disagreement $\hat a$, and optionally the jitter-sensitivity cue, and it outputs $T_i$.
During training, source volumes are resampled to thicker slices, their intensities are remapped, and they are given simulated noise and bias fields. This teaches the calibrator how shifted inputs look. The key constraint is that the soft-label target $a_i$ is always taken from the clean multi-rater annotations, so the target itself never shifts.
The third part is the compute accounting. Everything added runs on slabs after one 3D pass, so the full cost should stay well below a 5-member 3D ensemble. We report peak GPU memory and wall-clock time per volume alongside every accuracy number. The file lists volumetric ensembles as prohibitively expensive in clinical pipelines, so the cost comparison is part of the result, not a footnote.
Baselines
The naive reference is the frozen 3D U-Net's raw softmax. AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation reports that plain 3D U-Net often beats transformer variants on overlap. The second naive reference is a single global temperature fitted on source validation data. Both use the same trained backbone as every other method.
The strongest calibration competitor is the method of Ouyang et al., reimplemented with both its jitter cue and its denoising-autoencoder shape prior. It solves the same frozen-model, out-of-domain problem, so it is the method the proposal must beat. It gets the same simulated-shift training data and the same tuning budget as our calibrator. No reported result was found on Lacuna for that paper's calibration error figures, so we reproduce it ourselves. Beating it means lower expected calibration error (ECE, the gap between stated confidence and observed accuracy) and lower Brier score against rater soft labels on held-out scanners. It also means a higher area under the ROC curve (AUROC) for flagging cases whose Dice overlap drops sharply. We would like to see this especially on non-rigid structures, where its shape prior should fail.
The strongest uncertainty competitor is a 5-member deep ensemble of 3D U-Nets, reported both at full cost and at a matched compute budget.
The strongest multi-rater competitors are the Pionono and D-Persona models. For these we use their predicted rater spread directly as the uncertainty estimate. They are trained on identical multi-rater data with identical augmentation. No reported result was found on Lacuna for Q-Dice (the staged overlap metric used by the QUBIQ challenge) from these methods on QUBIQ, so all numbers are produced in-house under one protocol.
Three ablations isolate the key ingredient. The first swaps our geometry-based disagreement predictor for one that reads full-resolution raw intensities. The second trains the calibrator against consensus masks instead of soft votes. The third removes the slice-spacing input.
Why not just…?
Why not just use a deep ensemble or Monte Carlo dropout on the 3D model? This is a real threat on raw calibration, and ensembles may match us on ECE. Two things argue against it. First, an ensemble gives one uncertainty number per voxel with no way to say how much of it is expert ambiguity and how much is scanner shift. The failure-flagging test measures exactly this. Second, the file names full-volume ensembles as prohibitive in clinical pipelines. We report the ensemble at matched compute. If a matched-compute ensemble equals us on both calibration and failure flagging, the method has no reason to exist, and we will say so.
Why not just train a multi-rater model like Pionono or D-Persona with heavy intensity augmentation? If these models' predicted spread survives scanner shift, they already provide rater-grounded uncertainty. The measurement study in part one tests this directly. Our expectation that it fails rests on the file's observation that MRI intensity shifts cause drops above 20%, and these models read raw intensities end to end. If their spread holds up across scanners, we would narrow the contribution to the compute savings and the failure flag.
Why not just keep Ouyang et al.'s calibrator and drop the shape prior? That ablation is included. If the jitter cue alone performs as well, then learned disagreement adds nothing. The comparison on vessels and tumors is where the two should differ most.
Proposed Experiments
Multi-rater training and in-distribution evaluation use QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge, scored with its Q-Dice metric. Scanner shift is tested on multi-center MRI in two datasets. The first is Advances in automated fetal brain MRI segmentation and biometry: Insights from the FeTA 2024 challenge, whose held-out 0.55T low-field cohort is a severe hardware shift. The second is AMOS, whose CT-to-MRI and cross-vendor splits add further shifts.
Neither shift dataset is described in the file as having multiple raters. We will therefore collect three independent expert annotations on a modest subset of external cases, targeting on the order of 30 volumes per shifted domain. This gives a true soft-label target out of distribution. Without it, the separation claim cannot be tested.
Two experiments are decisive, and both must succeed. The first is that on the relabeled external cases, calibrated confidence tracks the external experts' soft votes better than Ouyang et al. and the matched-compute ensemble. The second is that the volume-level excess score $e$ predicts large Dice drops better than total entropy does. If $e$ only matches total entropy, the separation of human and domain uncertainty is not real.
A third check resamples AMOS MRI to progressively thicker slices and tracks calibration error as spacing grows, with and without the spacing input. A fourth reports memory and runtime per volume against the ensemble. Together these checks settle whether the geometry-based disagreement map is worth adding to a clinical pipeline.
References
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Diversified and personalized multi-rater medical image segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Multi-rater Prompting for Ambiguous Medical Image Segmentation
- Contextual embedding learning to enhance 2D networks for volumetric image segmentation
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
- Advances in automated fetal brain MRI segmentation and biometry: Insights from the FeTA 2024 challenge