Where Annotators Disagree Because of the Scanner: Separating Slice-Thickness Ambiguity from Anatomical Ambiguity in Multi-Rater 3D Segmentation

Most 3D medical scans are anisotropic: the voxels are much thicker along the slice axis than within each slice. A clinical MRI voxel can be 1×1×5 mm. When an organ boundary runs roughly parallel to the slices, as at the top and bottom of the prostate, each thick voxel holds a mixture of organ and background. The image then has no edge at that boundary, only a slow intensity ramp. Expert annotators place their contour at different points on that ramp, and they disagree. Current models of annotator disagreement treat it as one undifferentiated quantity, a "rater style" plus noise, learned on 2D images. They cannot say which part of the disagreement comes from the anatomy itself and which part comes from the acquisition, and only the second part would go away with a thinner-slice scan.

This note proposes a multi-rater segmentation model that contains an explicit model of how the scanner averages tissue across a slice. Each annotator is modeled as applying a personal threshold to a partial-volume image that the network predicts. The model then makes a sharp, testable prediction: on boundaries facing the slice axis, disagreement should widen in proportion to slice thickness, and on boundaries facing within the slice it should stay flat. A model of this kind can be trained at one slice thickness and should predict annotator disagreement at another. It also produces a per-voxel split of uncertainty into a part caused by acquisition and a part caused by the anatomy. Clinically, that split is the difference between "rescan" and "ask a second expert."

Core Claim

In thick-slice volumes, a large and predictable share of inter-observer disagreement comes from rater thresholds being spread out by the blur across each slice. It is not ambiguity in the anatomy. A model that makes this mechanism explicit will (a) match the multi-rater label distribution as well as flexible probabilistic models at the slice thickness it was trained on, (b) transfer to unseen slice thicknesses where those models fail, and (c) attribute disagreement to the right cause on data where the cause is known.

The motivating observation comes from Deep Learning Enables Prostate MRI Segmentation. That large-cohort study found that model–radiologist disagreement concentrates at the prostate base and apex and attributed it to partial-volume averaging across axial slices. However, it modeled the problem with a deterministic coronal-then-axial cascade and output a single mask. The multi-rater models that do represent disagreement have no notion of slice geometry. Pionono gives each rater a Gaussian in an 8-dimensional latent space whose mean encodes over- or under-segmentation bias. The Latent Doctor Model learns rater style from small support sets and conditions on the image. Multi-rater prompting adds about 0.1M rater-prompt parameters to a frozen U-Net. The variational inference model of inter-observer variability uses a 6-dimensional latent space. All four were evaluated on 2D images (pathology tiles, fundus photographs, 2D MRI slices). There, any anisotropy is invisible.

The method we borrow for the acquisition side comes from PV-SynthSeg. It simulates thick slices by blurring high-resolution label maps with anisotropic kernels and subsampling, and it trains a 3D network to recover isotropic segmentations, holding Dice 0.83 at 9 mm spacing. It uses deterministic labels and has no annotators.

The Model

The network predicts a high-resolution occupancy field $u(\mathbf{x}) \in [0,1]$ on an isotropic grid. It also predicts a latent anatomical variable $a$, which captures genuine uncertainty about where the organ is, independent of the scanner. Given the known slice profile $h$ (the averaging kernel along $z$ for that acquisition), the partial-volume fraction a rater sees in thick voxel $k$ is $$f_k(a) = \int h(z - z_k)\, u(x,y,z; a)\, dz.$$ Rater $r$ labels the voxel foreground when $f_k(a) > \tau_r$. The threshold $\tau_r$ is drawn from a per-rater distribution: its mean is the systematic bias, as in Pionono's latent mean, and its spread is intra-rater inconsistency.

The mechanism follows from one first-order consequence. Suppose thresholds vary across raters with spread $\sigma_\tau$. Then the spatial width of disagreement along the boundary normal $\mathbf{n}$ is approximately $$w \approx \frac{\sigma_\tau}{|\partial f / \partial \mathbf{n}|} + w_a,$$ where $w_a$ is the width contributed by variation in $a$. Because $h$ is wide along $z$, $|\partial f/\partial \mathbf{n}|$ falls roughly in inverse proportion to slice thickness for boundaries whose normal points along $z$. The acquisition term therefore grows linearly with thickness on those boundaries and stays constant on in-plane-facing ones. The anatomical term $w_a$ should not depend on thickness at all.

The model is trained on multi-rater labels with a per-rater likelihood. Rater-specific threshold parameters are cheap: a scalar mean and spread per rater, optionally image-conditioned through a small encoder. This addresses the cost problem that QUBIQ reports for 3D ensembles. Synthetic thickening in the style of PV-SynthSeg supplies extra training pairs at multiple thicknesses, because the forward model tells us exactly how the labels should change.

The model also takes a position on a tension in the boundary-loss literature. Losses such as the overlap loss in the left-atrium work push predictions toward confident, sharp transitions. That work reports Dice rising from 82.01% to 89.83% on a single-consensus benchmark. Our model keeps the sharpness inside $u$ and puts the softness where the physics puts it, in $f$. That lets us test whether sharpening losses help or hurt calibration on axial-facing boundaries.

Research Program

The first component is a 3D implementation of the model above on a standard 3D U-Net backbone. The backbone outputs $u$ at isotropic resolution, and a differentiable slice-profile layer computes $f$ for the acquisition's known slice thickness and spacing.

The second component is a controlled annotation study, because existing multi-rater data cannot separate the two causes. We take volumes with near-isotropic or thin-slice acquisitions (prostate MRI and abdominal CT are the target anatomies), synthetically thicken each to roughly 1, 3 and 5 mm, and have several annotators contour every version in randomized order. Within each case the anatomy is fixed and only the acquisition changes. Any change in disagreement with thickness is therefore attributable to acquisition. This gives ground truth for the decomposition.

The third component is a fully synthetic sanity check. Simulated raters with known thresholds annotate PV-SynthSeg-style thickened labels, and we verify that the model recovers $\sigma_\tau$ and a near-zero anatomical term.

The fourth component evaluates on the 3D QUBIQ tasks with clinical multi-rater labels: prostate MRI, kidney CT, pancreas CT and pancreatic lesion CT. We first confirm the actual voxel spacings of these volumes and analyze disagreement as a function of boundary orientation.

Baselines

The naive reference is a 3D U-Net trained on the majority-vote or STAPLE consensus, with softmax probabilities read as uncertainty. It shows how much of the multi-rater distribution a consensus model misses.

The strongest competitor for matching rater distributions in 3D is a deep ensemble. QUBIQ found that ensembles of deterministic models track rater distributions best across its 10-threshold Q-Dice, with top teams' average Dice between 0.80 and 0.85. The file gives no per-task number for the 3D QUBIQ tasks; no reported result found on Lacuna for QUBIQ prostate or pancreas per-task Q-Dice. On brain growth, the variational model reports continuous Dice 0.5372 against 0.5083 for Bayesian dropout, and we will reproduce that setting as a reference point. The ensemble gets the same backbone, augmentation and data, and its member count is set so that total training GPU-hours match our model plus its synthetic thickening.

The strongest structured competitor is a 3D port of Pionono, which reported $\kappa = 0.758$ on Gleason 2019 in 2D. We port it in two variants: without slice information, and with slice thickness given as an extra input. Both use the same backbone and receive the same hyperparameter search budget as our model.

The partial-volume baseline is PV-SynthSeg-style soft occupancy, thresholded at the empirical spread of rater thresholds but with no per-rater model. It tests whether physics alone, without rater modeling, is enough.

The key ablation removes the slice-profile layer, so the per-rater thresholds act directly on a thick-voxel probability map. This isolates whether the explicit forward model is what produces transfer.

Beating the competition means three things. At the training thickness, Q-Dice and per-rater Dice must be within noise of the ensemble. When trained at one thickness and tested at another, the error in predicted disagreement width must be clearly lower than the ensemble and both Pionono variants. And in the annotation study, the acquisition-attributed fraction of disagreement must correlate with the observed thickness-driven change.

Why not just…?

Why not just train a 3D ensemble or 3D Pionono with slice thickness as an input? This is the real threat. At a single thickness, such a model may match ours or beat it, and we expect that. The difference shows up out of distribution. A flexible model has to learn the thickness dependence from whatever thicknesses it sees. Ours has the linear-in-thickness scaling built in through $h$. The train-at-3 mm, test-at-1 and 5 mm experiment from the annotation study settles this. If the thickness-conditioned Pionono transfers equally well, the explicit forward model adds no predictive value, and the contribution reduces to the decomposition analysis.

Why not just predict partial-volume fractions, PV-SynthSeg style, and call the soft map the disagreement? The soft map captures where boundaries are ambiguous but not how individual raters resolve the ambiguity. Pionono and the Latent Doctor Model both show that systematic rater bias matters; the latter reports KL 0.461 and up to 0.728 Spearman correlation with rater entropy. The partial-volume baseline above tests directly whether rater thresholds add anything beyond the fraction map. If they do not, that is itself a useful finding: disagreement on thick slices would then be mostly physics.

Why not super-resolve the volume first and then run a 2D multi-rater model? The low-depth-resolution EM study found that classifying super-resolved images (F-measure 88.68%) underperformed classifying the low-resolution patches directly (90.07%) on FIB data, at extra cost. A super-resolved image also hides the acquisition origin of the ambiguity, which is exactly what we want to expose. We include super-resolution plus 2D Pionono as a secondary comparison on the annotation-study data only.

Why not slice in another plane, as the DANN cascade does? The DANN study used a coronal pass only to localize the gland's start and end slices. Disagreement still peaked at base and apex, because raters contour on the acquired axial slices regardless. Reslicing does not create information the acquisition never recorded.

Proposed Experiments

The first decisive check uses the annotation study. Plot observed disagreement width against slice thickness, separately for boundary patches whose normals point along $z$ and those whose normals lie in-plane. The hypothesis predicts roughly linear growth for the former and a flat line for the latter. If both are flat, or both grow equally, the mechanism is wrong and the proposal should be dropped. Second, on the same data, the model's per-voxel acquisition share of uncertainty should predict the per-voxel reduction in disagreement between the 5 mm and 1 mm versions, scored by rank correlation. The ensemble's total variance and the Pionono variants serve as comparators. Third, the transfer experiment trains every method at one thickness and measures Q-Dice, per-rater Dice and disagreement-width error at the others. Fourth, on the QUBIQ 3D tasks, we report Q-Dice at native resolution and the orientation analysis, checking whether axial-facing boundaries in pancreas and prostate account for the extra 3D difficulty that QUBIQ reported.

A further analysis adds the overlap loss of the left-atrium work to the consensus baseline. We then measure whether it worsens calibration specifically on axial-facing boundaries while improving consensus Dice, which is the trade-off the model predicts. Compute, latency and peak memory are reported for every method, since the scalable rater parameterization only matters if it beats ensembles on cost at equal fidelity.