Orthogonal Views as a Built-In Test for Whose Uncertainty It Is

When a segmentation model is unsure about a voxel in a CT or MRI scan, there are two very different explanations. The boundary may be genuinely ambiguous, so that experienced radiologists would also draw it in different places. Or the scan may come from an unfamiliar scanner or protocol, so the model's features are corrupted and its confidence means little. Clinically these call for different actions. The first calls for a second reader. The second calls for distrusting the model, or reacquiring the scan. Current methods produce one uncertainty map that mixes the two. The research file names this conflation as an open problem: rater-modeling methods assume that predictive entropy mirrors expert disagreement, while out-of-domain calibration work shows that entropy becomes unreliable under scanner shift.

We propose to separate the two using geometry that 2.5D models already have. A 2.5D model segments a 3D volume by running 2D networks on slices along the three perpendicular anatomical planes (axial, coronal, sagittal) and merging the results. Every voxel therefore receives three predictions from three different cuts through the same tissue. If each plane network is trained to reproduce the spread of expert opinion, a genuinely ambiguous boundary should look equally uncertain from all three directions, because ambiguity belongs to the anatomy. Scanner shift behaves differently. It comes with plane-dependent artifacts, anisotropic slice thickness, and interpolation. It should make the three views disagree with each other. The average uncertainty across planes becomes an estimate of rater ambiguity, and the disagreement between planes becomes a detector for shift that needs no training. We also want this to work when multi-rater labels are sparse, because paying several experts to trace whole volumes is exactly the bottleneck that keeps multi-rater 3D datasets small.

Core Claim

Consider a voxel $v$. Let $p_v^k$ be the foreground probability that plane network $k \in \{a, c, s\}$ predicts there, let $\bar p_v$ be the mean of the three, and let $H$ be binary entropy. The total predictive entropy splits exactly into two terms:

$$ H(\bar p_v) \;=\; \underbrace{\tfrac{1}{3}\textstyle\sum_k H(p_v^k)}_{A_v:\ \text{within-plane ambiguity}} \;+\; \underbrace{\Big[H(\bar p_v) - \tfrac{1}{3}\textstyle\sum_k H(p_v^k)\Big]}_{D_v:\ \text{cross-plane disagreement}} $$

Here $D_v$ is the Jensen–Shannon divergence among the three views. This identity is the same one used to split ensemble uncertainty into an aleatoric part and an epistemic part. What is new is the claim about the members. When the members are orthogonal views trained on soft multi-rater targets, $A_v$ tracks human disagreement and stays stable under scanner shift, while $D_v$ rises with shift. Ordinary ensemble members, or copies of the same view under test-time augmentation, share one view of the scanner physics, so their errors under shift are correlated and $D_v$ would miss the shift.

Research Program

The model has three 2D encoder–decoders, one per plane. They start from 2D pretrained weights, following the transfer argument in Deep Convolutional Neural Networks for Computer-Aided Detection. Each is trained with a soft cross-entropy loss against the fraction of raters who marked each pixel. This targets the rater distribution directly, rather than a majority-vote mask of the kind that A deep learning model integrating FCNNs and CRFs for brain tumor segmentation and Multi-branch convolutional neural network for multiple sclerosis lesion segmentation use in their late fusion. Those papers treat cross-plane disagreement as noise to vote away. We keep it as a signal.

We add an in-domain agreement loss on unlabeled source voxels, with weight $\lambda$. It pulls the three planes toward each other on familiar data, so that disagreement at test time carries more meaning. The loss is related to the uncertainty-weighted fusion in Uncertainty-aware multi-view co-training. That work uses view disagreement to weight pseudo-labels for adaptation. We use it as the output quantity itself, and we fit the rater distribution rather than a single ground truth. Too large a $\lambda$ risks making the three planes into one function that agrees even out of domain. For that reason $\lambda$ is swept, not fixed.

Raw $D_v$ is nonzero on in-domain data because of legitimate anisotropy. We therefore turn it into a shift score by comparing each case against the distribution of $D$ on held-out source cases, stratified by distance to the boundary. This step requires no target labels or source data at deployment, in keeping with the privacy constraints the file describes.

The label-efficiency arm asks whether the decomposition survives sparse supervision. Following Orthogonal Annotation Benefits Barely-supervised Medical Image Segmentation, each rater labels only a few perpendicular slices. The plane networks are supervised directly on the slices in their own plane, and the agreement loss spreads information to the other planes. Where DeSCO relies on deformable registration to propagate labels, we avoid it, borrowing the registration-free multi-view regularization of PA-Seg. The multi-rater target is available only where raters' slices overlap or intersect. We simulate this regime by subsampling dense multi-rater masks, which also gives us exact ground truth for evaluation.

Baselines

The naive reference is a single axial-only 2D network trained on soft rater targets. Its entropy is used as the only uncertainty signal, and its maximum softmax probability is used as the shift score. This is what most pipelines would do by default.

The strongest competitor for modeling rater ambiguity is a deep ensemble with the same entropy decomposition. The QUBIQ challenge (QUBIQ) reports that ensembles of deterministic models often beat probabilistic architectures, with top teams reaching "average Dice scores between 0.80 and 0.85". We will run three-member 3D and three-member 2D-axial ensembles. They get the same data, the same number of forward passes as our three plane networks, and the same tuning budget, and they are evaluated on the same QUBIQ tasks.

For rater modeling within a single network, we compare against the variational latent model of Variational Inference for Quantifying Inter-observer Variability. It encodes observer ambiguity in a 6-dimensional latent space and reports a continuous Dice of 0.5372 on QUBIQ brain growth, against 0.5083 for Bayes dropout. Our ambiguity map beats it if it matches its continuous Dice in domain and holds its correlation with rater entropy under shift.

For shift calibration, the reference is Improved post-hoc probability calibration for out-of-domain MRI segmentation. It combines perturbations applied at test time with a denoising-autoencoder shape prior and reports expected calibration error falling from 14.48% (local temperature scaling) to 12.78% on simulated artifacts, and from 16.78% to 15.37% on the bSSFP→LGE cardiac shift. We apply its perturbation score to our model's outputs, so that only the shift signal differs between the two conditions. We win if calibration with $D_v$ reaches at most 15.37% expected calibration error on bSSFP→LGE, without a shape model or extra augmented passes.

For the sparse arm, the reference is DeSCO, which reports 86.93% Dice on KiTS19 from 10 slices. KiTS19 has a single rater, so we compare Dice only.

The key ablation replaces the three orthogonal planes with three axial networks trained from different seeds. Everything else stays the same: soft targets, agreement loss, and compute. This isolates orthogonality as the ingredient. If this variant detects shift as well as the orthogonal version, the core claim fails.

Why not just…?

Why not just use a deep ensemble's mutual information as the shift signal? It is the most direct threat, and the same-view-seeds ablation above settles it. The case for orthogonal views is that ensemble members see identical inputs, so a plane-structured artifact corrupts all of them the same way. The file reports that QUBIQ found volumetric ensembles expensive in 3D and that The Latent Doctor Model calls one-model-per-expert ensembles prohibitive. Neither point shows that ensembles detect shift poorly, so we treat this as an open empirical question and do not assume the answer.

Why not just use test-time augmentation, as the post-hoc calibration paper does? Reslicing along other planes could be seen as a structured form of augmentation. Unlike brightness jitter, though, it probes the physical anisotropy of the acquisition. It also has a limitation: that paper's shape prior reportedly struggled with irregular structures, and was evaluated only on cardiac boundaries. We will run intensity augmentation both alone and combined with $D_v$. If the combination adds nothing over augmentation alone, the note will say so.

Why not just use an evidential head that separates "no evidence" from "conflicting evidence"? Multi-view Evidential Learning-based Medical Image Segmentation places a Dirichlet output on fused features. Its uncertainty describes the model, not measured rater spread, and it was tested only on fully labeled 2D data. We will train per-plane Dirichlet heads as an alternative decomposition. If their evidence-strength term detects shift as well as $D_v$ does, then plane geometry is unnecessary.

Why not just train a 3D U-Net with heavy augmentation? AMOS shows a 3D U-Net reaching 89.04% mean Dice on CT but losing over 20% on MRI from unseen scanners. It also gives no separation between the two kinds of uncertainty. We include it as an accuracy reference only.

Proposed Experiments

The decisive experiment uses controlled shift on multi-rater data. On the QUBIQ prostate MRI, brain-growth MRI, kidney CT, and pancreas CT tasks, we apply simulated MRI artifacts of increasing severity, plus resampling to thicker slices, to the test volumes. Applying synthetic shift to real multi-rater data gives us two known ground truths at once: rater disagreement is fixed by the labels, and shift severity is set by us. We measure two things across severity levels. The first is the Spearman correlation between $A_v$ and the per-voxel entropy of the raters, compared with the 0.728 correlation between model and pathologist entropy that the Latent Doctor Model reports for ISUP grading. The second is the area under the ROC curve (AUROC) for detecting shifted versus clean volumes using the mean of $D_v$.

The prediction that could fail is this. As severity grows, the correlation for $A_v$ should degrade much less than the correlation for total entropy, and $D_v$ should increase steadily with severity. We also check the reverse leak: $A_v$ must not rise on shifted scans at voxels where the raters agreed.

Natural shift is tested on AMOS MRI from scanners held out during training, and on ACDC bSSFP→MS-CMRSeg LGE. We report shift-detection AUROC, voxel-error detection, and expected calibration error against the numbers listed in the Baselines section. We also report Q-Dice in domain, to confirm that splitting uncertainty does not cost accuracy relative to the ensembles.

The sparse arm repeats the QUBIQ experiments, keeping only one to three orthogonal slices per rater per volume. We plot the ambiguity correlation and shift AUROC against the label budget, down to the budget DeSCO uses. We also sweep $\lambda$ to locate where agreement training stops sharpening $D_v$ and starts suppressing it.

The result that would sink the proposal is the same-view-seeds ablation matching the orthogonal model on shift detection under both synthetic and natural shift. That outcome would mean the benefit comes from ensembling alone, and the paper would report it as such.