Models that must work when some inputs are missing, such as a brain-tumor segmenter that sometimes gets all four MRI sequences and sometimes only one, are usually trained by randomly hiding modalities. When a full-modality "teacher" is available, they are also trained to copy that teacher's answers. Copying can backfire. If the only sequence that shows a structure is hidden, then demanding the student reproduce the teacher's answer for that structure asks it to predict something it cannot see. The student either memorizes or hallucinates. This proposal borrows an older and less used idea from learning with privileged information: data seen only during training should explain how hard each example is, not supply targets to imitate. Here, the modalities that are hidden from a training example become that example's privileged information. A small training-only network reads them and decides, voxel by voxel and subset by subset, how much each error made from the available inputs should be penalized.
Stated in two sentences: we train one segmentation network on random modality subsets, plus a throwaway helper that sees the full scan and reweights the loss to forgive errors that are unavoidable given what was hidden. The bet is that this makes the deployed model better and less prone to hallucination on the hard subsets, for example enhancing tumor without contrast-enhanced T1. The helper is discarded at test time, so inference cost is unchanged.
Core Claim
Learning using privileged information (LUPI) originated with SVM+ (A new learning paradigm: Learning using privileged information). In SVM+, a correcting function on the privileged features predicts each example's slack, meaning how much margin violation it is allowed. Learning using privileged information: SV M+ and weighted SVM relates this to example weighting and reports faster convergence when the slack is well estimated. The deep-learning descendants of LUPI mostly dropped this "explain the difficulty" mechanism in favor of imitation. Generalized distillation (Unifying distillation and privileged information David Lopez-Paz) and the missing-modality distillers built on it ask the student to match a privileged teacher.
Our claim is about which tool fits missing modalities. When a subset $S$ of modalities lacks the information needed to identify part of the label, imitation is the wrong form of privileged supervision and slack is the right one. Concretely, we predict three outcomes from a privileged-slack loss on a standard incomplete-modality backbone:
- It beats both plain modality-dropout training and full-teacher distillation on subsets missing the dominant modality.
- It produces fewer false-positive structures on those subsets.
- It cannot be replicated by a difficulty estimate computed from the available inputs alone.
Relation to Prior Work
Three lines of work already join privileged learning and missing modalities. All three use imitation.
Disentangled-Multimodal Privileged Knowledge Distillation for Depression Recognition with Incomplete Multimodal Data (Dis2DR) trains a complete audio-visual-text teacher. It distills disentangled shared and modality-specific features into an incomplete student. Its own ablations show performance still collapses when text is missing, which is the failure our method targets.
Graph Distillation for Action Detection with Privileged Modalities makes the single deployment modality imitate a sample-weighted mixture of privileged modalities. It assumes one fixed test modality rather than arbitrary subsets.
Scratch Each Other's Back: Incomplete Multi-modal Brain Tumor Segmentation Via Category Aware Group Self-Support Learning (GSS) assigns hand-chosen "leader" modalities per tissue and distills their soft labels into the other branches. This is closest to us in setting and in having zero inference overhead. It still pushes weak branches toward the leader's answer, and it relies on manual clinical priors.
We keep the setting of these papers and replace "match this answer" with "this error is forgivable to this degree, given what you could not see." Distilled Prompt Learning for Incomplete Multimodal Survival Prediction also treats missing genomics as privileged. It does so through a frozen language model, which does not carry over to dense volumetric MRI.
Two papers supply the mechanism. Deep Learning Under Privileged Information Using Heteroscedastic Dropout has a privileged network set the variance of noise injected into a student's features. We move that idea from feature noise to per-voxel loss weights and condition it on which subset is observed. Coupled Training with Privileged Information and Unlabeled Data supplies a constraint. On UCI Parkinson's, two-stage distillation with noisy privileged features raised MSE from 77.34 (plain supervised) to 83.32, while coupled joint training reached 72.60. We therefore train the slack head jointly with the main network rather than in a separate stage, and we test robustness to corrupted privileged modalities directly.
The generative line is a competitor, not a source. That line includes Unbiased Missing-modality Multimodal Learning, which documents asymmetric "generation bias", and Calibrated Multimodal Representation Learning with Missing Modalities, which imputes missing representations to correct "anchor shift". Both fill in what is missing. We never fill in anything at test time.
Research Program
The deployed model $f$ is an existing incomplete-modality segmenter. We use the RFNet backbone from RFNet: Region-aware Fusion Network for Incomplete Multi-modal Brain Tumor Segmentation so that comparisons inherit a well-tuned baseline. It is trained, as usual, on random modality subsets $S$ drawn from the full set $\mathcal{M}$ of four MRI sequences.
Next to it sits a lightweight slack head $g$ that exists only during training. It receives the full set of modalities $x_{\mathcal{M}}$ and a binary mask encoding $S$. It outputs a per-voxel log-slack map $s(x_{\mathcal{M}}, S)$. The training loss on one example is
$$ \mathcal{L} = \sum_{v} w_v\, \ell_v\big(f(x_S), y\big) + \beta \,\mathrm{Var}_v(s_v), \qquad w_v = \frac{e^{-s_v}}{\frac{1}{V}\sum_{u} e^{-s_u}}, $$
with the following terms:
- $\ell_v$ is the usual per-voxel segmentation loss.
- $V$ is the number of voxels.
- $w_v$ is the weight on voxel $v$, normalized to average one per example.
- $\beta$ penalizes extreme reallocation.
The normalization is the key design choice. It stops the head from discounting the loss everywhere, which is the "give up on hard cases" failure of learned-variance losses. The head can only move attention away from some voxels and onto others.
The head never sees labels, and it is trained jointly with $f$. It can therefore only learn which voxels the subset model predictably gets wrong given full knowledge of the scan. That is the deep, per-voxel analogue of SVM+ predicting slack from the privileged space. We expect the head to learn, for example, that enhancing-rim voxels visible only in contrast-enhanced T1 deserve low weight when that sequence is hidden. Edema boundaries visible in FLAIR should keep high weight whenever FLAIR is present.
A variant separates the difficulty caused by the missing modalities from difficulty that is intrinsic to the scan. It trains a second head $g_S$ that sees only $x_S$ and uses the residual $s - \mathrm{sg}(s_S)$ as the slack, where $\mathrm{sg}$ stops gradients. We report this variant as an analysis tool and keep the simpler head as the primary method unless the variant clearly wins.
Two analyses accompany the method. The first visualizes the learned weight maps against tissue classes and subsets. It checks that the forgiven regions match known modality sensitivities without our having encoded those priors, the way GSS does by hand. The second measures, per subset, the volume of predicted enhancing tumor in scans whose ground truth has none. We call this the hallucination rate. Imitation-based methods should raise it on subsets lacking contrast-enhanced T1, and privileged slack should lower it.
Baselines
The naive reference point is RFNet trained with random modality dropout and nothing else. The file reports its 15-combination average Dice on BraTS 2020 as 86.98% whole tumor, 78.23% tumor core and 61.47% enhancing tumor (RFNet). We retrain it on the same split to remove pipeline differences.
The second baseline is full-teacher distillation on the same backbone. A frozen RFNet trained with all modalities provides soft labels, and the subset model matches them alongside the task loss, following Unifying distillation and privileged information David Lopez-Paz. A Dis2DR-style feature-matching term is added as a second configuration. This baseline is the direct test of imitation against slack.
The strongest existing method is GSS on the RFNet backbone (Scratch Each Other's Back). It reports average Dice gains of 1.27% to 3.20% over RFNet across BraTS 2015, 2018 and 2020, and up to 8.52% on enhancing tumor in BraTS 2020. The file gives no absolute GSS Dice values, so no reported absolute result was found on Lacuna for GSS on BraTS 2020. We rerun it.
All methods share the following controls:
- the same backbone, augmentation, epoch count and subset-sampling distribution;
- the same hyperparameter search budget of twelve trials each over learning rate and the method's own coefficient (distillation weight, GSS masking rate, or our $\beta$);
- three seeds;
- an identical deployed network, since our head and GSS's branches are both discarded.
Beating GSS means a higher mean enhancing-tumor Dice over the 15 combinations, outside seed variance, with no loss on the complete-modality input. A lower hallucination rate on subsets lacking contrast-enhanced T1 is the secondary criterion.
Three ablations isolate the key ingredient. The first is a self-slack head that sees only $x_S$, which removes privileged information while keeping the reweighting machinery. The second is a shuffled-privileged head that receives another patient's full scan, which removes sample-specific information while keeping access to modality statistics. The third drops the mask input, so the head no longer knows which subset it is explaining. A focal-loss run, which reweights by current difficulty with no extra head, rounds out the comparisons.
Why not just…?
Why not just distill the complete-modality teacher, as Dis2DR and GSS do? This is the main alternative, and the experiments are built to decide it. The argument against imitation is specific: on a subset missing the only sequence that reveals a structure, the teacher's soft label is not recoverable from the input. Matching it rewards guessing. Dis2DR's own sharp drop when text is missing (Dis2DR) and the negative transfer in Coupled Training with Privileged Information and Unlabeled Data are consistent with this, though neither isolates it. If distillation matches privileged slack on enhancing-tumor Dice and hallucination rate for subsets without contrast-enhanced T1, the claim is wrong. We would then report that imitation suffices here.
Why not just let the model predict its own per-voxel uncertainty from the inputs it has? This is the most serious threat. In principle, the expected difficulty of a voxel given $x_S$ can be estimated from $x_S$ itself. The self-slack ablation is exactly this alternative, run under the same normalization. Our hypothesis is that the full scan carries sample-specific information about where the hidden modality would have changed the answer, and that $x_S$ cannot supply it. This is the same reason SVM+ uses privileged features to estimate slack instead of the decision features. If self-slack matches privileged slack, the contribution reduces to "normalized loss reweighting helps." We would state that plainly.
Why not just impute the missing modality or its representation? Imputation methods such as CalMRL and MD2N fill the gap rather than forgive it. MD2N's own finding is that cross-modal generation is asymmetric and biased. In segmentation, a synthesized contrast-enhanced T1 is precisely where an enhancing rim could be invented. Imputation also adds inference cost, which our method does not. We do not claim imputation is dominated in accuracy. We include a representation-level imputation baseline in the hallucination-rate comparison, because that is where the two approaches should differ most.
Why not just rely on modality dropout with a strong fusion architecture? mmFormer and Gradient-Guided Modality Decoupling for Missing-Modality Robustness show that architecture and gradient handling matter, and GMD reports a 5.3% enhancing-tumor gain on BraTS 2018. Our loss is orthogonal to the backbone. As a secondary experiment, we add it to mmFormer to check that the gain is not specific to RFNet.
Proposed Experiments
BraTS 2020: primary benchmark. We evaluate all 15 modality combinations on Dice for whole tumor, tumor core and enhancing tumor. We additionally report expected calibration error per subset and the hallucination rate defined above. The decisive check is a three-way comparison, on the seven subsets lacking contrast-enhanced T1, of privileged slack against full-teacher distillation and against self-slack. Our prediction is that privileged slack wins enhancing-tumor Dice and hallucination rate against both, while matching them on complete input. The weight-map analysis should show forgiven regions concentrating on enhancing tumor when contrast-enhanced T1 is hidden, without any prior telling the head this.
Robustness to bad privileged data. We corrupt the privileged copy of one sequence with noise or misregistration in a fraction of training scans. Coupled training on Parkinson's suggests imitation degrades below the naive baseline under such corruption (Coupled Training). We predict privileged slack degrades only toward the dropout baseline, because the weights are normalized and the head never sets prediction targets.
Non-random missingness. Current benchmarks drop modalities at random. We add a test split in which contrast-enhanced T1 is missing preferentially for scans with small enhancing regions, mimicking the selection effects discussed in Correcting for selection bias and missing response in regression using privileged information. This tests whether the gains survive non-random missingness.
AVEC 2013: transfer to a different task. On depression recognition from audio, video and text, we compare against Dis2DR under its protocol and focus on the text-missing case, where imitation fails worst. Dis2DR reports up to a 9.8% MAE/RMSE improvement over AVA-DepressNet. No absolute AVEC 2013 MAE for Dis2DR was found on Lacuna, so we rerun it. For a regression target, the slack is a per-sample weight rather than a voxel map.
Kill criteria. The idea is abandoned if either of two results appears on BraTS 2020. The first is that self-slack and shuffled-privileged slack are within seed noise of privileged slack. The second is that full-teacher distillation matches it on both enhancing-tumor Dice and hallucination rate for subsets lacking contrast-enhanced T1.
References
- A new learning paradigm: Learning using privileged information
- Learning using privileged information: SV M+ and weighted SVM
- Unifying distillation and privileged information David Lopez-Paz
- Disentangled-Multimodal Privileged Knowledge Distillation for Depression Recognition with Incomplete Multimodal Data
- Graph Distillation for Action Detection with Privileged Modalities
- Scratch Each Other's Back: Incomplete Multi-modal Brain Tumor Segmentation Via Category Aware Group Self-Support Learning
- Distilled Prompt Learning for Incomplete Multimodal Survival Prediction
- Deep Learning Under Privileged Information Using Heteroscedastic Dropout
- Coupled Training with Privileged Information and Unlabeled Data
- Unbiased Missing-modality Multimodal Learning
- Calibrated Multimodal Representation Learning with Missing Modalities
- RFNet: Region-aware Fusion Network for Incomplete Multi-modal Brain Tumor Segmentation
- mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation
- Gradient-Guided Modality Decoupling for Missing-Modality Robustness
- Correcting for selection bias and missing response in regression using privileged information