When several experts trace the same brain structure on an MRI scan, they disagree, and recent models learn to reproduce that disagreement instead of collapsing it into one "true" mask. Almost all of these models are trained and tested one structure at a time: a single region, often on 2D slices, with nothing modeled around it. Brain structures, however, tile the brain. The hippocampus has no free edge. Every voxel a rater takes from the hippocampus goes to the amygdala, the ventricle, or white matter. Our claim is that inter-rater disagreement in brain segmentation is mostly disagreement about where shared borders sit. A model scoped to one structure therefore counts each border twice, treats the two counts as independent, and gets wrong the uncertainty of anything computed from more than one region: volume ratios, asymmetry indices, or the fraction of a lesion that falls in each territory.
We propose a measurement study and a model. The study collects multi-rater delineations of a medial temporal neighborhood and shows whether neighbor-volume disagreement is strongly negatively correlated, which single-structure models cannot represent. The model is a rater-style latent model whose output is a single partition of the neighborhood. A rater's style therefore moves a shared border as one decision. The model's scope is "a structure plus everything it touches," a deliberately bounded middle ground between single-ROI models and whole-brain parcellation, which QUBIQ reports is computationally prohibitive for multi-rater 3D uncertainty.
Core Claim
Let $V_A$ and $V_B$ be the volumes a rater assigns to two adjacent structures. Any quantity built from both depends on their covariance across raters:
$$\mathrm{Var}(V_A + V_B) = \mathrm{Var}(V_A) + \mathrm{Var}(V_B) + 2\,\mathrm{Cov}(V_A, V_B).$$
If disagreement mostly concerns the shared border, $\mathrm{Cov}(V_A,V_B)$ is strongly negative. The union is then stable while the ratio $V_A/V_B$ is much less stable than the marginals suggest. Per-structure uncertainty models effectively set the covariance to zero. They overstate uncertainty for unions and understate it for ratios and boundary-dependent scores. A second problem is that their samples are not partitions: voxels get claimed by two structures or by none.
The claim is falsifiable on two counts. If measured cross-rater covariance between neighbors turns out near zero, the premise fails. If a model scoped to a structure and its neighbors does no better than a post-hoc fix of per-structure samples, the modeling contribution fails.
Where this sits in prior work
Two lines of work model rater disagreement directly, and both supply our method while defining our gap. Liu et al. encode rater interpretations in a 6-dimensional latent space trained with a modified evidence lower bound, and sample plausible alternative masks on the single-structure QUBIQ brain-growth task. Pionono learns a Gaussian latent distribution per rater, whose mean captures that rater's systematic bias and whose variance captures their inconsistency, on histopathology. We reuse Pionono's per-rater latent almost unchanged. What we change is the output: one categorical partition over a neighborhood rather than one binary mask per structure.
The QUBIQ benchmark (Li et al.) constrains the design in two ways. It found that plain deep ensembles often beat elaborate probabilistic models at matching multi-expert distributions, so ensembles are our most serious competitor. It also notes the cost of extending such models to full 3D volumes, which is why we bound the scope at neighborhoods instead of attempting whole-brain coverage.
On the coverage side, SLANT handles whole-brain 133-label segmentation by splitting space into 27 overlapping tiles with separate 3D U-Nets, fused by majority vote. That work shows spatial scoping can be made compute-feasible, but it models no rater uncertainty, and its labels come from multi-atlas silver standards plus 45 manual OASIS scans.
Deep-ASPECTS motivates the downstream stakes. It computes a stroke score by intersecting an infarct mask with 10 separately segmented territories, and reports only final-score agreement (76% within 2 points of radiologists). It never separates territory-border disagreement from lesion disagreement. The single-region critiques in the file, such as the left-hippocampus-only morphometry in GeoHSAF, motivate asking what is lost when the scope is one structure.
FeTA 2024 supplies an evaluation lens. Its finding that good overlap scores hide topological defects tells us to check whether sampled segmentations remain anatomically valid, measured with Euler characteristic difference (a count of spurious holes and disconnections).
Research Program
The first component is data, because no dataset in the file has multi-rater, multi-structure 3D brain labels. On 30 OASIS T1 scans, five trained raters will independently delineate a medial temporal neighborhood: hippocampus, amygdala, entorhinal cortex, the temporal horn of the lateral ventricle, and adjacent white matter as a background class. All raters follow a single protocol in which the neighborhood is traced jointly.
On a 10-scan subset, the same raters will also trace the hippocampus alone under a single-structure protocol, after a washout interval. This tests the human side of the scope question: does tracing a structure in isolation change where and how much raters disagree? Each scan will receive one repeat trace from each rater, which lets us measure how inconsistent each rater is with themselves. The dataset is modest but sufficient to estimate neighbor covariances. Its value lies in the joint labels, not the scan count.
The second component is analysis. Across rater pairs, we will map where disagreement voxels fall, distinguishing shared borders from borders with background. We will also estimate the full cross-rater covariance matrix of structure volumes and test whether neighbor covariances are significantly negative.
The third component is the model. A 3D U-Net receives the neighborhood crop, which is placed using an affine-registered atlas prior. Following Pionono, the network is conditioned on a sample $z \sim q(z \mid r)$ from rater $r$'s latent distribution. It outputs a single softmax over all neighborhood labels, so every sample is a partition by construction. For new scans with no rater identity, we sample from the mixture of rater distributions.
The crop size is the experimental variable we call scope. We compare the single structure, the structure plus its touching neighbors, and a SLANT-size tile. This tells us how much context the uncertainty actually needs.
The fourth component is propagation. Sampled segmentations feed three downstream quantities: hippocampal volume, the hippocampus-to-amygdala volume ratio, and left-right hippocampal asymmetry. On ADNI scans we compare the width of the resulting intervals against the AD-versus-control group difference in each quantity. If rater-level border uncertainty in the ratio approaches that group difference, single-structure models are misleading users about the reliability of these biomarkers.
Baselines
The naive reference point is a single deterministic 3D U-Net trained on majority-vote labels, with uncertainty taken from softmax entropy. It shows what you get by ignoring raters entirely.
The first strong competitor is per-structure probabilistic modeling: the Liu et al. variational model and Pionono, each trained separately per structure on its own crop. This is how these methods are currently used. On QUBIQ brain growth, Liu et al. report a continuous Dice of 0.5372, against 0.5083 for Bayes Dropout and 0.5042 for ESULR. We will first reproduce that number to confirm our implementation before using it on our data.
The strongest competitor is a deep ensemble of multi-class neighborhood U-Nets, one member per rater plus bootstrap members. This baseline matters because QUBIQ found ensembles outperform complex probabilistic models, with top teams averaging Dice between 0.80 and 0.85 across tasks. An ensemble of multi-class models already outputs partitions, so it directly threatens our modeling claim. No reported result exists in the file for any method on multi-rater, multi-structure 3D brain segmentation.
All methods share the same backbone family, crops, augmentation, training data, and a matched GPU-hour budget. Ensemble size is set so that its total compute equals the latent model's training plus sampling cost. Each method receives the same hyperparameter search budget, tuned on a held-out rater fold.
Two ablations isolate our key ingredient. The first keeps the same latent model but replaces the shared softmax with independent per-structure sigmoid heads. The second keeps the shared softmax but varies scope from single structure to neighborhood to tile.
Our method beats the ensemble only if, at matched compute, it does better on both held-out-rater coverage and covariance error for the ratio quantity. A gain on distribution metrics such as Q-Dice alone would not count.
Why not just…?
Why not just fuse raters with majority vote or STAPLE? Fusion produces one mask, so it cannot predict the spread of a held-out rater's volumes at all. Wu et al. and Pionono both report that fusion discards clinically relevant rater variation; Pionono reaches a kappa of 0.758 on Gleason 2019 against STAPLE-based alternatives. In our setting the naive baseline makes this concrete: its coverage of held-out rater values should collapse wherever disagreement is real.
Why not just train a multi-class deep ensemble on the neighborhood? This is the real threat. If ensemble members trained on different raters already reproduce the negative neighbor covariance, then the scope finding stands but our latent model adds little. The matched-compute comparison settles it. We expect the ensemble to capture between-rater bias but to underrepresent each rater's inconsistency with themselves, which the repeat traces let us measure. If it does not, we will report the ensemble as the recommended tool and keep the scope analysis as the contribution.
Why not keep per-structure models and fix their samples afterward, by giving each voxel to the structure with the highest probability? That repair restores a partition, but the coupling between neighbors it produces is an accident of independent sampling rather than learned rater behavior. The independent-heads ablation with this repair applied tests whether that is good enough. The diagnostic is whether its neighbor covariance and ratio coverage match the shared-softmax model.
Why not argue the borders are thin and per-structure uncertainty is good enough? That is an empirical question, and the first analysis answers it directly. If disagreement voxels fall mostly at borders with background rather than at shared borders, or if neighbor covariances are near zero, the premise is wrong and we will say so.
Proposed Experiments
The first decisive check is the covariance measurement on the new annotations. We report the fraction of disagreement voxels lying on shared borders and the estimated neighbor volume covariances with bootstrap intervals over scans.
The second check uses leave-one-rater-out evaluation. For each method we compute the 90% interval it predicts for each downstream quantity and ask how often the held-out rater's value falls inside. A well-calibrated method should be near 90%. Alongside coverage we report:
- the error between predicted and observed covariance matrices;
- the partition-violation rate in samples, meaning the fraction of voxels claimed by zero or by more than one structure;
- Euler characteristic difference of samples at low and high probability thresholds, following FeTA 2024;
- Q-Dice for comparability with QUBIQ.
The low-threshold topology check matters because thresholding soft borders at 0.1 is exactly where adjacent structures merge.
The third check is the human scope comparison on the 10-scan subset: is inter-rater disagreement on the hippocampus different under isolated versus joint tracing?
The fourth check is the ADNI propagation analysis: how wide the ratio and asymmetry intervals are relative to the AD-versus-control group difference.
As an external sanity check, all methods will also be run on QUBIQ brain growth, to confirm that our implementations of the single-structure baselines match their reported numbers.
References
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- 3D whole brain segmentation using spatially localized atlas network tiles
- Deep-ASPECTS: A Segmentation-Assisted Model for Stroke Severity Measurement
- GeoHSAF: Geometric Hippocampus Shape Analysis Framework for Longitudinal Alzheimer's Disease Classification
- Advances in automated fetal brain MRI segmentation and biometry: Insights from the FeTA 2024 challenge
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle