When several radiologists outline the same tumor or organ, they almost always agree on where the structure is and what its core looks like. They disagree in a thin band at the edge: one draws tight margins, another draws generous ones, and a third drifts outward only where the tissue is ambiguous. Current models of this disagreement ignore that structure. They regenerate the entire mask for every rater or every sample, either from a latent code injected at the network's bottleneck or from a separate decoder per rater. We propose a different model. A shared network first predicts one consensus boundary as a signed distance map, meaning each pixel stores how far it is from the boundary, with the sign telling inside from outside. A second, much lighter stage then predicts, only within a narrow band around that boundary, how far each rater would move the boundary along its normal. That displacement is predicted as a probability distribution over a small set of candidate offsets rather than as a single number.
The design is borrowed from coarse-to-fine depth estimation. There, a coarse stage fixes the global geometry, and finer stages search only a narrow interval of depth values around the coarse estimate. Read that way, the depth axis becomes the boundary-normal axis, and the candidate depth planes become candidate boundary positions. The depth literature also supplies a warning. If the coarse stage commits to a single answer, any true surface that falls outside the fine search band is lost for good. Keeping several hypotheses alive fixes this. We claim three things: inter-observer variability is captured at least as well by a spatially varying, multimodal offset field on a shared scaffold as by full-mask generators; it is captured at a fraction of their compute; and the representation directly reports, in millimetres, where and by how much experts disagree.
Core Claim and How It Differs From Prior Work
Two lines of medical work model rater disagreement, and both regenerate whole masks. The first uses latent codes. Pionono (Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation) gives each rater a distribution in an 8-dimensional latent space that conditions a U-Net decoder. Variational Inference for Quantifying Inter-observer Variability in Segmentation of Anatomical Structures samples a 6-dimensional latent code to produce plausible masks. The second uses dedicated decoders. The Multi-decoder U-Net and OMBA-UNet (Inter-Rater Uncertainty Quantification in Medical Image Segmentation via Rater-Specific Bayesian Neural Networks) assign a full decoder to each rater. These papers motivate our proposal and serve as its main competitors.
Two of them also report the costs of the full-mask approach. OMBA-UNet's latency rises from 0.02 s to 0.34 s per slice. The Multi-decoder U-Net generalizes poorly on brain growth, where data per rater branch is sparse. We do not condition the whole mask on the rater. Rater-specific computation is confined to boundary-band pixels, and its output is a displacement, not a class label.
A third medical method constrains our output design. MoSE (Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts) shows that annotations are multimodal and that averaging them yields blurry outputs. It addresses this with several stochastic decoders and an optimal-transport loss, but its decoder is flat and single-scale. We keep multimodality without mixture decoders by making each offset prediction a categorical distribution.
The depth papers supply the mechanism. Cascade cost volumes (Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching) and cost volume pyramids (Cost Volume Pyramid Based Depth Inference for Multi-View Stereo) search narrow residual depth intervals around a coarse prediction. The cascade paper reports halving GPU memory with this approach. Residual pyramids for single-image depth (Structure-Aware Residual Pyramid Network for Monocular Depth Estimation) and for filling in sparse depth (FCFR-Net) refine a coarse base map with additive corrections. FCFR-Net also reports the matching failure mode: errors in the coarse map degrade the refined result.
Non-parametric Depth Distribution Modelling based Depth Inference for Multi-view Stereo supplies the fix for early commitment. It carries the top several depth hypotheses per pixel through the pyramid and reports a 32% error reduction at depth discontinuities. Its sparse 3D convolutions cost about 1.2 s per scene, so we adapt the idea in a cheaper form: a categorical distribution over a few offset bins on the boundary band. Finally, Robust Depth Completion with Uncertainty-Driven Loss Functions combines coarse-to-fine refinement with predicted uncertainty. Its uncertainty is sensor noise, which is a single-mode quantity, whereas ours is a distribution over human annotators that can have several modes. No work in the file joins coarse-to-fine refinement with inter-observer modeling.
Research Program
Let $\phi_0(x)$ be the consensus signed distance at pixel or voxel $x$, predicted by a standard segmentation backbone trained against the signed distance of the per-pixel median rater mask. Let $\tau$ be a band half-width in millimetres. Within the band $|\phi_0(x)|<\tau$, a light refinement head predicts a probability vector $p_\theta(k \mid x, c)$ over $K$ fixed offset bins $d_1,\dots,d_K$ spanning $[-\tau,\tau]$. Here $c$ is either a rater identity embedding, when raters are consistent across cases, or a sampled mode index, when they are not. Rater $r$'s boundary is then the zero level set of
$$\phi_r(x) = \phi_0(x) - d_{k_r(x)}, \qquad k_r(x) \sim p_\theta(\cdot \mid x, c_r),$$
where $k_r(x)$ is the offset bin selected for rater $r$ at $x$. Training minimizes the negative log-likelihood of each rater's observed offset bin. That observed offset is measured by sampling the rater's true signed distance along the consensus normal. A total-variation penalty along the boundary keeps the offset field coherent rather than speckled; this targets the pixel-noise failure that MoSE and Pionono describe for latent samplers.
When identities are unavailable, as with the anonymized-style multi-reader sets, we use $M$ learned mode embeddings. During training each rater is assigned to its best-matching mode with a minimum-over-modes loss, a lightweight relative of MoSE's distribution-matching loss.
The program has three stages. The first is a diagnostic study, run before any model is built. On every dataset we measure the distribution of rater offsets from the median boundary. This tells us whether disagreement really sits in a narrow band, which sets $\tau$. It also measures how much of the variance is explained by a single per-rater scalar margin and how much requires a spatially varying field. We also measure the "coverage loss": the fraction of rater boundary points lying outside the band, which is the segmentation analogue of early commitment. The second stage builds the 2D and 3D model with the band head. The third stage tests whether the predicted per-location offset spread, in millimetres, matches the observed inter-rater spread. That match is the clinically usable output for margin planning and for checking radiomic sensitivity.
Baselines
The naive reference is a standard U-Net trained on the averaged soft label, thresholded at multiple levels. Because QUBIQ scores agreement with the averaged rater map, this "do nothing" model is a real competitor. If our method does not beat it on Q-Dice, the extra structure is not earning its keep on that metric. Next to it we run a consensus model trained on majority vote or STAPLE masks. This is the clinical default, and Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle argues it discards useful variance.
The strongest existing competitors depend on the benchmark. On QUBIQ brain growth, the best number in the file is a continuous Dice of 0.5372 for the variational model of Liu et al. (Variational Inference for Quantifying Inter-observer Variability in Segmentation of Anatomical Structures), against 0.5083 for Bayes dropout. On the other QUBIQ tasks, the Multi-decoder U-Net reports Dice of 0.852 and 0.871 on brain tumor tasks 1 and 2, 0.870 on kidney, and 0.553 on prostate task 1. The QUBIQ challenge summary reports top averages of 0.80–0.85 for ensembles of deterministic models.
On LIDC-IDRI, MoSE and OMBA-UNet both report lower generalized energy distance than the Probabilistic U-Net and PHiSeg. However, no reported GED value for MoSE or OMBA-UNet was found on Lacuna, so we will rerun both from their published descriptions. OMBA-UNet's reported latency of 0.34 s per slice is our compute reference.
All methods share one backbone family, the same splits and augmentation, and the same hyperparameter search budget, measured in GPU-hours. Ensembles are reported both at matched training compute and at their natural size.
Beating the baselines means two things. First, our method must match or exceed the best full-mask method on Q-Dice, continuous Dice, and GED. Second, it must give a clear improvement on rater-specific boundary error (per-rater 95th-percentile Hausdorff distance) and on correlation with the observed millimetre spread. It must do both at substantially lower inference cost than OMBA-UNet or an ensemble.
Three ablations isolate the key ingredients. A scalar-margin model gives each rater one learned dilation or erosion of the consensus distance per case. A flat model regresses each rater's signed distance directly, with no consensus scaffold. A single-mode band head regresses the mean offset instead of a distribution over bins. We also sweep $\tau$ to trace the trade-off between coverage loss and accuracy.
Why not just…?
Why not just train a deep ensemble? The QUBIQ summary shows ensembles lead the challenge. It also names their computational cost as the main barrier and calls for compact single-model alternatives. Ensembles also do not model rater identity: their spread reflects training randomness, which need not resemble the spread between experts. We will run ensembles at matched compute and at natural size. If an ensemble of equal cost matches us on GED and on spread correlation, the efficiency argument fails, and we will report it.
Why not just dilate or erode the consensus mask per rater? This is the most serious threat. If experts differ mainly by a global conservatism, which is what Pionono's latent mean is meant to capture, then one scalar per rater on $\phi_0$ is a nearly free model. Our diagnostic study measures how much offset variance a scalar explains, and the scalar-margin ablation tests it end to end. If the scalar model is within noise of the full band head, we will narrow the contribution. The scaffold-plus-margin representation would still beat full-mask generators on cost and interpretability, but the spatially varying, multimodal field would be shown to be unnecessary on these datasets.
Why not just use Monte Carlo dropout? Dropout reflects the model's uncertainty about its own weights, not the distribution of experts. It scores 0.5083 continuous Dice on brain growth against 0.5372 for variational modeling, and MoSE reports that such sampling produces unstructured pixel variance. We include it as a cheap reference, but it is not the main competitor.
Why not just use a latent-code model like Pionono with an added consensus loss? Global latent injection could, in principle, learn to leave the interior untouched. Our flat-versus-scaffold ablation, run with matched latent dimension, tests whether explicitly restricting the rater effect to the band improves sample coherence and per-rater Hausdorff error, especially on low-agreement brain growth, where per-rater data is sparse.
Proposed Experiments
We evaluate on QUBIQ (prostate, brain growth, brain tumor, and kidney tasks, with 3 to 7 raters), LIDC-IDRI (4 raters), and the identity-aligned liver lesion set used by OMBA-UNet, where rater-conditioned predictions can be scored directly. Metrics are Q-Dice and continuous Dice on QUBIQ, GED and calibration error on LIDC-IDRI, per-rater Dice and 95th-percentile Hausdorff distance where identities are known, Pearson correlation between predicted and observed per-location spread in millimetres, coverage loss, and latency per slice and per volume.
Three checks are decisive. The first is the scalar-margin comparison, which decides whether spatial structure in disagreement is real. The second is the single-mode versus categorical band head, measured on boundary segments where raters split into separate groups. If the distribution adds nothing there, the borrowed defence against early commitment is unnecessary for human raters. The third is the $\tau$ sweep: if accuracy only saturates once the band covers most of the structure, the narrow-band premise is false. If all three come out in our favour and compute is several times lower than OMBA-UNet, the result shows that inter-observer variability is better treated as a boundary refinement problem than as a mask generation problem.
References
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures
- Uncertainty Quantification in Medical Image Segmentation with Multi-decoder U-Net
- Inter-Rater Uncertainty Quantification in Medical Image Segmentation via Rater-Specific Bayesian Neural Networks
- Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts
- Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
- Cost Volume Pyramid Based Depth Inference for Multi-View Stereo
- Structure-Aware Residual Pyramid Network for Monocular Depth Estimation
- FCFR-Net: Feature Fusion based Coarse-to-Fine Residual Learning for Depth Completion
- Non-parametric Depth Distribution Modelling based Depth Inference for Multi-view Stereo
- Robust Depth Completion with Uncertainty-Driven Loss Functions
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge