When a brain tumor MRI arrives without one of its usual sequences, such as the contrast-enhanced T1 scan that shows where the blood-brain barrier has broken down, a segmentation model's boundary becomes less certain. That uncertainty has two different sources that call for different clinical responses. Part of it is disagreement that experts would show even with every scan in hand; the answer there is wider safety margins or a second reader. The other part is information that the missing scan would have supplied; the answer there is to acquire the scan. Today these two sources are studied by separate communities. Missing-modality networks are scored against a single consensus mask. Multi-rater networks, which model how different experts draw boundaries, assume every input channel is present. Neither can say which kind of uncertainty it is reporting.
We propose a model that reports both terms separately for each voxel, using the law of total variance. The first term is the expected disagreement among raters given the full scan. The second is the spread in the full-scan answer that the missing sequence would have resolved. We pair this with a small human re-annotation study, in which the same experts contour the same tumors with and without particular sequences. This gives the first ground truth for how much human disagreement actually grows when a modality is withheld. The study is what makes the idea testable. Without it, a model's "missing-information" uncertainty cannot be checked against anything clinicians would recognize.
Core Claim
Let $x_o$ be the observed sequences, $x$ the full four-sequence scan, and $y$ a rater's binary label at a voxel. Let $p(x)$ be the fraction of raters who label that voxel as tumor given the full scan. Then
$$\mathrm{Var}(y \mid x_o) = \underbrace{\mathbb{E}\big[p(x)(1-p(x)) \mid x_o\big]}_{\text{expert disagreement}} + \underbrace{\mathrm{Var}\big(p(x) \mid x_o\big)}_{\text{information lost with the missing sequence}}.$$
This is the law of total variance applied to a Bernoulli label, conditioned first on the full scan and then on what was observed.
The central observation is simple, and it cuts against an obvious design. If you train a multi-rater model with random modality dropout on soft rater labels, it already learns the correct pixelwise marginal $\mathbb{E}[p(x)\mid x_o]$. So "a joint missing-modality multi-rater network" is not a meaningful contribution by itself; it is our main baseline. What that combination cannot do is split the total into the two terms. We claim three things. First, the split can be learned from full-modality training data with a teacher–student setup. Second, the second term predicts which cases actually change when the missing sequence is supplied. Third, the first term, not the total, matches human disagreement on the full scan, while the second matches the increase in human disagreement when a sequence is withheld.
The claim has a corollary for evaluation. Scoring a missing-modality model with Q-Dice against full-scan rater maps penalizes the correct, broader prediction. The benchmark protocol itself therefore needs to change.
Research Program
The teacher is a full-modality multi-rater segmenter. We use a shared encoder with per-rater decoders, in the style of the Multi-decoder U-Net, which trains one decoder per annotator with a cross-loss so the branches share information. The teacher gives, for every training case, per-rater maps whose mean estimates $p(x)$ and whose spread estimates the disagreement term.
The student is a missing-modality encoder. It uses per-sequence encoders fused by permutation-invariant pooling as in HeMIS, or cross-sequence attention with modality-level dropout as in mmFormer. Its input is a randomly masked version of the same scan. The student has three outputs. The first is a mean map regressed onto the teacher's $p(x)$. The second is a disagreement map regressed onto the teacher's rater spread. The third is a residual map trained to predict the squared error between its own mean and the teacher's $p(x)$. That squared-error target is the standard way to estimate a conditional variance when only one full-scan realization exists per case. Because training scans are complete, every target is available without any synthesis.
Our design differs from teacher–student distillation in Robust Divergence Learning, which distills a full-modality teacher into a partial-modality student with Hölder pseudo-divergence and mutual information. That work distills a single consensus boundary. We distill a distribution plus an explicit residual, so the student learns what it cannot know, not only what the teacher knows. We keep Hölder divergence as an optional loss ablation, since that paper reports a 6.1% gain over KL for missing-modality segmentation. We treat it as a method source, not a claim.
The architecture question is secondary. We will run the decomposition heads on a HeMIS-style backbone and on a UniMRSeg backbone (UniMRSeg, code at Xiaoqi-Zhao-DLUT/UniMRSeg). If the result only holds on one backbone, it is not about the decomposition.
The data plan combines two sources. We pretrain the student's encoders on BraTS 2018/2020 consensus labels with modality dropout. We then fit the teacher and the decomposition heads on the multi-rater brain tumor tasks of QUBIQ (3 to 7 raters per scan) and on the multi-rater subset of BRATS 2012/2013 (65 cases, up to four raters). We will verify which sequences each QUBIQ brain task provides and restrict the masking experiments to those.
The human study is the part no prior work has. Four neuroradiologists re-contour about 30 cases in three conditions: full scan, all sequences except T1c, and FLAIR only. The order is randomized and a washout of several weeks separates conditions. Each rater also repeats a full-scan subset, which gives an intra-rater noise floor. The study directly measures how much human disagreement grows when a sequence disappears, and where in the tumor the growth happens. Menze et al. motivate the prediction that it concentrates in the enhancing core: they report enhancing-core agreement only in the high 50s to low 60s even with all sequences, and mmFormer reports that the enhancing tumor collapses without T1c.
Baselines
The naive reference point is a full-modality multi-rater model with zeroed missing channels. It is the Multi-decoder U-Net trained on complete QUBIQ inputs and evaluated with missing sequences set to zero. This is how existing multi-rater systems would actually be deployed on an incomplete scan. On complete inputs this model reports Dice 0.852 on QUBIQ brain tumor task 1 and 0.871 on task 2 (Multi-decoder U-Net). No reported result was found on Lacuna for any multi-rater model under missing sequences. Beating it is a sanity check, not a result.
The strongest direct competitor is the obvious combination: the same student backbone trained with modality dropout on soft multi-rater labels, whose predictive variance $\mu(1-\mu)$ is reported as a single uncertainty. It gets the same data, backbone, epochs and hyperparameter search budget as our model. The only difference is the absence of the decomposition heads. This is the ablation that isolates the key ingredient. For the consensus-segmentation side, the relevant reported figure is UniMRSeg's mean whole-tumor Dice of 80.64% on BraTS 2020 averaged over modality subsets, against 76.39% for PASSION (UniMRSeg). We require our mean head to stay within a pre-registered 1 Dice point of the same backbone trained without our heads. The decomposition must not cost segmentation accuracy.
The strongest method for the information term specifically is diffusion multiple imputation. It uses an adversarial diffusion translator (SynDiff), or the faster wavelet flow-matching synthesizer (WFM), to sample K plausible versions of the missing sequence. Each version runs through the full-modality teacher, and the variance of the teacher's mean across samples is the information term. This baseline estimates exactly our second term by brute force, so it is a real competitor. We will give it the same training data and report its inference cost. Beating it means matching or exceeding its correlation with the human study's increase in disagreement at a fraction of its runtime. If it wins clearly on accuracy, we will report that and frame our contribution as the cheap approximation.
A deep ensemble with a standard epistemic/aleatoric split is also included, trained under modality dropout with five members.
Why not just…?
Why not just train the multi-rater model with modality dropout and report its total uncertainty? The total variance is exactly what our first baseline outputs, and for pixelwise Q-Dice it may be all one needs. The question is whether the split carries information the total does not. The test is concrete. Among cases missing T1c, predict which ones change enhancing-core Dice by more than a fixed amount once T1c is restored, computed directly from BraTS full scans. We compare AUROC for total variance against AUROC for our information term. If total variance does as well, the decomposition is not worth its heads. We will say so, and the contribution reduces to the human study and the evaluation protocol.
Why not just use ensembles or Monte Carlo dropout to split epistemic from aleatoric uncertainty? That split separates uncertainty about model weights from noise in the data. Missing-sequence uncertainty is not about the weights: with infinite training data it remains, because the signal is physically absent. So it lands in the aleatoric bucket, mixed with rater disagreement. The ensemble baseline tests this directly. If its epistemic term tracked the human study's increase in disagreement, our premise would be wrong.
Why not just synthesize the missing scan and run a full-modality multi-rater model? A single synthesized image collapses the missing information to one guess. The model then reports only expert disagreement, confidently, on possibly hallucinated enhancement, which is the failure UniMRSeg describes as error propagation. Sampling many syntheses fixes the collapse and is our third baseline. The remaining argument against it is cost and the fidelity of rare pathology in samples, both of which we measure rather than assume.
Why not just evaluate on consensus labels, as mmFormer and UniMRSeg do? A consensus mask erases the first term entirely. The Latent Doctor Model shows consensus training discards expert uncertainty, reporting entropy correlation with a 20-pathologist panel up to 0.728 when that uncertainty is modeled instead. Scoring against a consensus mask would make our two heads unidentifiable.
Proposed Experiments
The first decisive check uses the human study. For each withheld-sequence condition, we compute the voxelwise increase in human rater variance over the full-scan condition, net of the intra-rater floor. We then correlate it with the model's information term, predicted from that same restricted input. Separately, we correlate full-scan human variance with the model's disagreement term. Success means the information term correlates with the increase in disagreement more strongly than the disagreement term does, and more strongly than total variance does. The pattern should reverse for full-scan disagreement. If both heads track the same human quantity, the decomposition has failed.
The second check is the acquisition-decision AUROC described above. It is computed on held-out BraTS 2020 cases, where restoring the sequence is free because the full scan exists.
The third check re-scores all models with Q-Dice and generalized energy distance under two targets. One target is the full-scan rater maps, as QUBIQ does today. The other is the human study's restricted-condition rater maps. We expect the ranking of missing-modality models to change between the two targets. A change in ranking would show that current practice mis-scores them. Every comparison is run across all subsets of the available sequences, and reported per tumor region, since whole-tumor results can hide the enhancing-core behavior the idea is about.
References
- Uncertainty Quantification in Medical Image Segmentation with Multi-decoder U-Net
- HeMIS: Hetero-Modal Image Segmentation
- mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation
- Robust Divergence Learning for Missing-Modality Segmentation
- UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
- The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)
- Unsupervised Medical Image Translation With Adversarial Diffusion Models
- WFM: 3D Wavelet Flow Matching for Ultrafast Multi-Modal MRI Synthesis
- The Latent Doctor Model for Modeling Inter-Observer Variability