When several radiologists outline the same brain lesion, their outlines differ. A growing family of models is trained to reproduce that spread, so that a model can report "experts would disagree here" instead of a single overconfident mask. These models are always trained and scored on images and labels that have already passed through a fixed preprocessing pipeline. That pipeline resamples voxels to a standard grid, corrects intensity drift, strips the skull and may warp the brain to a template. This note proposes to measure how much of the uncertainty these models predict, and how much of the human spread they are scored against, is actually created or erased by those preprocessing choices rather than by the anatomy.
The core idea is to treat the preprocessing pipeline as a second, controlled source of disagreement and compare it, voxel by voxel, to the disagreement between human raters. If swapping one reasonable pipeline for another moves predicted boundaries by about as much as swapping one expert for another, then current multi-rater benchmarks are confounded. In that case, uncertainty models need to be trained and evaluated in a way that holds up across pipelines. If the pipeline effect is small, we will have extended a known negative result about preprocessing to the uncertainty setting. Both outcomes are worth reporting.
Core Claim
Two literatures currently miss each other. Preprocessing studies score models against one fixed label. Negligible effect of brain MRI data preprocessing for tumor segmentation varied eight pipelines for nnU-Net and UNETR and found the following:
- Only voxel resampling clearly mattered.
- Skull-stripping helped moderately on two cohorts.
- Bias correction, denoising and histogram matching had no significant effect on Dice.
That study treated the manual masks as deterministic truth. Multi-rater studies, in turn, all inherit a single pre-curated preprocessing. This is true of the variational model of Variational Inference for Quantifying Inter-observer Variability, the rater-specific decoders of OMBA-UNet, and the confusion-matrix disentangling of Disentangling Human Error from Ground Truth. The QUBIQ challenge fixed preprocessing for all entrants by design.
We claim that a Dice-neutral preprocessing change can still be uncertainty-relevant. It can move the soft probability map at exactly the ambiguous boundaries where raters disagree, while leaving the confident interior untouched. Average Dice would not see this, but uncertainty metrics would.
To make this testable, we define a boundary-band variance ratio. Let $p_{\pi}(v)$ be a model's predicted foreground probability at voxel $v$ under pipeline $\pi$, mapped back to the native acquisition grid. Let $y_r(v)$ be rater $r$'s label at $v$ on that same native grid. Let $B$ be the set of voxels where raters do not unanimously agree. Then
$$R = \frac{\sum_{v \in B} \mathrm{Var}_{\pi}\big[p_{\pi}(v)\big]}{\sum_{v \in B} \mathrm{Var}_{r}\big[y_r(v)\big]},$$
where the numerator is the spread across a set of reasonable pipelines and the denominator is the spread across raters. A value of $R$ near or above roughly 0.3 would mean pipeline choice is a first-order contributor to the uncertainty a model reports. This threshold is a working cutoff we will fix before running the study, not a derived quantity. A value of $R$ near zero would mean it is not.
Separately, we measure how Q-Dice changes when it is computed on resampled versus native grids. Q-Dice is QUBIQ's score that averages Dice over probability thresholds from 0.1 to 0.9. This comparison tests whether the human reference spread is itself altered by label interpolation.
Research Program
The work has three parts.
Measurement. We build a small, explicit grid of pipelines from operations the file documents as standard. The grid includes:
- isotropic resampling with nearest-neighbour, trilinear or B-spline interpolation of both image and labels;
- N4 bias correction on or off;
- denoising on or off;
- z-score versus histogram-matched intensities;
- two skull-stripping tools where raw heads are available (HD-BET and ROBEX, as used in the BraTS Toolkit);
- native space versus affine template space, with deformable registration as an additional arm.
For speed and for a learned-preprocessing arm, we use Neural Pre-processing. Its user-set smoothness weight lets us sweep preprocessing strength continuously without retraining. For each pipeline we train the same multi-rater uncertainty model. We then compute $R$, Q-Dice in native and resampled space, and generalized energy distance between sampled masks and the set of rater masks.
Method. If $R$ is large, the obvious fix is pipeline marginalization. We sample a pipeline per training case, and at test time we average predictions over several pipelines after mapping them back to native space. The goal is uncertainty that matches the human spread and stays stable across pipelines. All evaluation is done in the native acquisition space so that labels are never interpolated for scoring.
Rater sub-study. On a small subset, two or three raters re-annotate the same scans presented under two pipelines: bias-corrected versus raw, and native versus resampled. This tests whether preprocessing changes the human tracings themselves, a confound that Disentangling Human Error cannot see because its pipeline was constant.
Data and Constraints
We need multi-rater MRI labels, and preferably raw images. The candidate datasets are:
- The QUBIQ brain-growth task (seven raters per slice) and the QUBIQ brain-tumor task. These have pre-curated images, so only downstream steps (resampling, intensity normalization, bias correction) can be varied.
- MSLSC, a multiple sclerosis lesion dataset with multi-expert annotations.
- The FeTA 2024 fetal brain data (FeTA 2024 insights), where super-resolution reconstruction is itself a heavy preprocessing step and a human inter-rater reference exists for biometry.
The GBM, LGG and BGPD cohorts from the negligible-effect study supply raw multi-sequence scans for the skull-stripping and registration arms. They carry single labels, so on them we can only measure the numerator of $R$, and we will report it in absolute terms. The main constraint is that 3D multi-rater modeling is memory-bound, as QUBIQ documented. We therefore use a 2.5D backbone where full 3D does not fit, and we hold the backbone fixed across all arms.
Baselines
The naive reference is a single fixed pipeline: the dataset's own curation plus isotropic resampling. On top of it we train a deterministic segmentation network on the majority-vote mask and use its softmax probability as the uncertainty map. This is what most deployed systems do. Any proposal that cannot beat it on Q-Dice and generalized energy distance has no case.
The strongest existing competitor for matching human spread is a deep ensemble of deterministic networks. The QUBIQ summary reports that such ensembles often beat latent probabilistic models, with top entries reaching average Dice between 0.80 and 0.85. A second strong competitor is a latent-variable multi-rater model, the variational approach of Variational Inference for Quantifying Inter-observer Variability. On QUBIQ brain growth it reports a Continuous Dice of 0.5372, against 0.5083 for Bayes Dropout. A third is OMBA-UNet, for which the file reports wins on 5 of 7 QUBIQ tasks but no numeric value; no reported result found on Lacuna for its per-task scores. All of these will be retrained by us on the same splits, the same backbone capacity and the same number of GPU hours. Each gets a fixed hyperparameter search budget. Each ensemble member counts against the compute budget.
Beating these baselines has two parts. First, the method must reproduce or exceed 0.5372 Continuous Dice on brain growth under the original pipeline. Second, across the pipeline grid, the spread of its Q-Dice must be smaller than that of the ensemble and the variational model, with no loss in mean Q-Dice.
The key ablation isolates pipeline sampling. We compare it against nnU-Net-style intensity and spatial augmentation tuned to produce the same per-voxel input perturbation magnitude. A second ablation compares native-space against resampled-space evaluation, which isolates the label-interpolation effect from the model effect.
Why not just…?
Why not just use standard data augmentation? Intensity and spatial augmentation already expose a network to variation like bias fields and resampling blur. This is the most serious threat, and it may well match pipeline sampling. However, augmentation perturbs images while leaving labels on the same grid. It cannot reproduce the label-interpolation effect of resampling anisotropic rater masks, which is the part of the problem we expect to be specific to multi-rater work. The matched-magnitude ablation settles this. If augmentation closes the gap, we drop pipeline sampling as a method and report the measurement study alone.
Why not just ensemble models trained on different pipelines? This would likely capture pipeline variance. It is a natural variant of our marginalization and will be run as one. It costs one model per pipeline. QUBIQ already flagged ensemble cost as prohibitive in 3D. We compare it to single-model pipeline sampling at equal compute.
Why not just fix one pipeline and declare the problem standardized? Fixing the pipeline hides the variation rather than removing it. Glocker et al. found that scanners remain identifiable at 98.82% accuracy after a full standard pipeline, and that cross-site sex classification falls to 62.16%. FeTA 2024 attributes about 0.10 Dice to image quality and about 0.075 to site. Deployed models will meet pipelines other than the benchmark's. Measuring $R$ tells us whether that matters for uncertainty specifically.
Why not just disentangle annotator error post hoc? The confusion-matrix approach of Disentangling Human Error attributes all label deviation to raters. Because its pipeline was constant, it cannot separate a rater's habit from a tracing the rater adjusted in response to a clipped skull-strip or a contrast change. The rater sub-study directly measures whether such pipeline-induced tracing changes exist.
Decisive Checks
The first check is the sign and size of $R$ on QUBIQ brain growth, QUBIQ brain tumor and MSLSC. If $R$ is near zero everywhere, the method arm is abandoned and the result stands as an uncertainty-level extension of the negligible-effect finding.
The second check is whether Q-Dice of the strongest baseline changes by more than its across-seed variation when only the evaluation grid moves from resampled to native. If it does, current benchmark rankings partly reflect interpolation.
The third check is whether pipeline sampling beats matched augmentation on cross-pipeline Q-Dice stability.
The fourth check, on FeTA, compares predicted-uncertainty-derived biometry spread against the 5.38% MAPE human inter-rater reference. We test whether that spread changes with the super-resolution reconstruction settings.
References
- Negligible Effect of Brain MRI Data Preprocessing for Tumor Segmentation
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures
- Inter-Rater Uncertainty Quantification in Medical Image Segmentation via Rater-Specific Bayesian Neural Networks
- Disentangling Human Error from Ground Truth in Segmentation of Medical Images
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
- BraTS Toolkit: Translating BraTS Brain Tumor Segmentation Algorithms Into Clinical and Scientific Practice
- Neural Pre-processing: A Learning Framework for End-to-End Brain MRI Pre-processing
- Advances in automated fetal brain MRI segmentation and biometry: Insights from the FeTA 2024 challenge
- Machine Learning with Multi-Site Imaging Data: An Empirical Study on the Impact of Scanner Effects