Modeling Reader Disagreement in Depth Without Inventing Missing Slices

When several radiologists outline the same lung nodule or pancreatic lesion on a CT scan, they disagree. A useful model should reproduce that disagreement instead of averaging it away. Nearly all models that learn from multiple annotators work on single 2D slices. A CT scan, though, is a stack of slices, and much of what radiologists disagree about is three-dimensional: whether a lesion is present at all, on which slice it begins and ends, and how its boundary continues from one slice to the next. Clinical scans also do not sample depth evenly. Many are acquired with thick slices or gaps between them. Standard pipelines fill those gaps by interpolation, which makes guessed tissue indistinguishable from imaged tissue.

We propose a multi-annotator segmentation model with two properties. First, each simulated radiologist ("reader") is one random draw shared across the whole volume, not a fresh draw on every slice. Second, the model is told which slices were actually acquired and which positions lie in gaps, rather than being fed interpolated images. The claim is that reader disagreement is a property of the whole volume and depends on how the scan was acquired. It is largest along the depth axis and in the gaps between acquired slices. A model that respects both facts should match the real spread of expert outlines better than slice-wise generative models, isotropic-resampled 3D ensembles, or deterministic soft-label networks. It should also keep its calibration across groups of scans acquired differently.

Core Claim

We make two linked, testable claims. The first concerns volumetric context. Sampling a reader's "style" independently per slice, which is what 2D latent-variable models such as the Probabilistic U-Net and Pionono implicitly do when run over a stack, yields samples that switch readers between adjacent slices. Such samples cannot represent the most common 3D disagreement, which is where along depth a reader stops drawing. This is the multi-annotator version of the fragmentation that Roth et al. documented for deterministic 2D pancreas segmentation.

The second concerns informative missingness. Slice thickness and gaps are protocol choices, not random noise, so the absence of an acquired slice carries information. The tabular literature summarized in our file shows that imputing missing values and then classifying erases that information, and that explicit missingness indicators are at least as informative as imputation. We transfer that lesson to the depth axis. Interpolating to isotropic voxels is imputation, and an explicit "observed or interpolated" indicator per depth position is the alternative.

If both claims hold, readers' z-extent disagreement should grow with slice spacing in real data. A model given the indicator should place its spread there, and an interpolating model should not.

Relation to Prior Work

Several papers motivate the gap. QUBIQ benchmarked multi-rater uncertainty on 2D and 3D tasks. It reported that uncertainty quality degraded when moving to 3D, and that ensembles of deterministic U-Nets beat generative probabilistic models. That result constrains us: any latent-variable proposal must beat ensembles, not just other generative models. Re-thinking and Re-labeling LIDC-IDRI showed that reader subjectivity biases LIDC-IDRI labels, which it resolved by hard relabeling with 3D metric learning on nodule crops. We instead model the subjectivity, at the level of the scan rather than a cropped nodule.

Other work supplies methods we reuse. The Probabilistic U-Net injects a low-dimensional latent late in the network, which makes sampling cheap. The Hierarchical Probabilistic U-Net adds multi-scale latents, but only on 2D grids. Pionono and the Latent Doctor Model give each rater a distribution in latent space, and the Liu et al. variational model does the same for anatomical structures. All of these work per image or per tile. Our change is to share the latent across depth and to condition decoding on acquisition. Disentangling Human Error estimates per-annotator confusion matrices, including on BraTS volumes, but slice by slice and with no account of acquisition gaps.

On the missingness side, RoVTL and PaRaChute handle missing tabular columns or entire missing modalities with indicators and gating. Neither looks at missingness inside a volume. Missingness-Aware Conformal Prediction showed that pooled calibration hides subgroup under-coverage driven by missingness patterns, and fixed it with Mondrian (group-wise) calibration on the most-shifted missingness indicator. We use that method directly as our calibration lens, with acquisition spacing as the grouping variable.

Research Program

The model has four parts. The backbone is anisotropy-aware: it uses in-plane 2D convolutions with depth-wise mixing over native slices, and could use linear-cost state-space blocks in the style of Swin-UMamba for long stacks. It runs on a depth grid at fixed resolution. Grid positions that coincide with an acquired slice carry image intensities and an indicator $m_z = 1$. Positions in gaps carry no intensity and $m_z = 0$; they are never filled by interpolation.

A reader latent $u_r$, low-dimensional and Gaussian as in Pionono, is drawn once per reader per volume and injected at every depth position. A rater-identity prior handles datasets with consistent reader identities. A set-encoder prior over the observed masks, following the Latent Doctor Model, handles anonymized readers such as those in LIDC-IDRI. The decoder outputs a full 3D mask, including on gap positions, so the model is forced to represent uncertainty exactly where the image is silent. Training maximizes the usual evidence lower bound over each reader's 3D mask.

At inference, sampling $u$ gives coherent whole-volume hypotheses. Their spread yields per-lesion summaries: whether the lesion is present, its first and last slice, and its volume. We then calibrate prediction intervals for these summaries with split conformal prediction, both pooled and Mondrian-grouped by slice-spacing bin, following You et al..

An empirical study comes first, because the missingness claim depends on it. On LIDC-IDRI, whose scans vary in slice thickness, we measure how the spread of readers' first and last annotated slices, in millimetres, varies with slice spacing. We also measure how much of total 3D disagreement, by voxel count, sits at the depth extremes versus in-plane boundaries. If z-extent disagreement does not grow with spacing, the acquisition indicator has no clinical signal to capture. In that case the proposal reduces to the volume-coherent latent alone, and we will report it that way.

Baselines

The naive reference is a 3D nnU-Net-style network trained on majority-vote masks after isotropic resampling, with uncertainty read off softmax probabilities. It represents current practice in both of the respects we challenge.

The strongest existing competitor is a deep ensemble of deterministic 3D U-Nets, each trained on resampled volumes with one reader's masks. This is the approach that won QUBIQ. QUBIQ reports average Dice of 0.80 to 0.85 for top teams on its staged threshold metric (Q-Dice) across tasks. We found no reported Lacuna result for 3D pancreas or kidney CT specifically. We will match its total training compute, roughly five members against one latent model trained about five times as long, and give both the same hyperparameter search budget.

Two generative baselines complete the set. The first is the 2D Probabilistic U-Net run slice-wise with independent latents. The second is the 2D Hierarchical Probabilistic U-Net, whose paper reports reconstruction IoU of 0.97 versus 0.75 for the Probabilistic U-Net on 2D LIDC-IDRI crops. That number is not comparable to whole-volume evaluation, and no GED value is reported on Lacuna. We also include Pionono applied per slice, and a deterministic 3D soft-label network that regresses the per-voxel fraction of readers.

Two ablations isolate our ingredients. One shares the latent across depth but feeds interpolated slices, which isolates the indicator. The other keeps the indicator but samples latents per slice, which isolates volume coherence. Beating the ensemble means three things. The proposal must achieve lower 3D generalized energy distance (GED), a sample-set distance between predicted and real reader masks. It must achieve lower Wasserstein distance between predicted and real per-reader z-extent distributions on scans in the top spacing tercile. And it must not lose on Q-Dice, all at equal or lower inference cost.

Why not just…?

Why not just use a 3D deep ensemble on resampled volumes? This is the real threat, given the QUBIQ evidence. Ensembles do capture between-model diversity. However, members trained on different readers learn to produce consistent masks, and the Probabilistic U-Net paper found that ensembles collapse toward dominant modes. Because they are trained on interpolated slices, they also cannot tell which slices were guessed. The thick-slice stratum is where the experiment settles this. If the ensemble matches our z-extent and GED numbers there at equal compute, our architectural argument fails, and we will say so.

Why not just run a 2D Probabilistic U-Net and sample the latent once per volume? This is nearly our model without depth-wise mixing and without the indicator, and it is the cheapest serious alternative. The volume-coherence ablation includes exactly this configuration. If it matches the full model, the contribution reduces to "share the latent," a useful but small finding. We expect it to fall short at lesion ends, where 2D features cannot see that a structure is tapering.

Why not just regress per-voxel reader agreement with a deterministic 3D network? Soft-label regression may calibrate voxel-wise marginals well and will likely do fine on Q-Dice. It cannot, however, produce joint samples. It therefore cannot answer questions such as "how many readers would call this lesion present" or "what is the spread of lesion volume." Our sample-level metrics and conformal intervals on lesion volume are designed to expose this limitation. On marginal metrics alone, we expect a tie.

Why not just interpolate and add slice thickness as a scalar input? This weaker form of the indicator tells the model a scan is sparse but not where the gaps are. We run it as an additional ablation. The tabular evidence summarized by You et al. suggests that position-specific indicators matter. Whether that holds here is an empirical question.

Proposed Experiments

Datasets are LIDC-IDRI (four readers, variable slice thickness, whole-scan nodule masks) and the 3D QUBIQ tasks, namely pancreas CT, pancreatic lesion CT and kidney CT. For brain-growth MRI, the Liu et al. variational model reports a continuous Dice of 0.5372 against 0.5083 for Bayes dropout, which gives a 2D anchor.

The first decisive check is the empirical study above: does reader z-extent spread grow with slice spacing on LIDC-IDRI?

The second is a controlled subsampling test on thin-slice scans. We remove every second or third slice, keep the original thin-slice reader masks as targets, and ask whether the model's predicted spread rises specifically in the gaps while the interpolating ablation's spread does not.

The third is calibration. Pooled split conformal intervals for lesion volume and z-extent may reach nominal 90% coverage overall while under-covering the thick-slice group, mirroring the 77.5% subgroup coverage You et al. found in tabular data. Mondrian calibration by spacing bin should close that gap, and the remaining width of the intervals shows how informative each model's samples are.

The fourth is a cross-site test. We train on thin-slice-dominant LIDC sites, evaluate on thick-slice sites, and compare coverage degradation across the ensemble, the soft-label network and our model.

The proposal succeeds if the full model beats the compute-matched 3D ensemble on thick-slice GED and z-extent distance and does not lose on Q-Dice. It is refuted if the shared-latent 2D ablation or the ensemble closes that gap.