Normative modeling is one of the main ways neuroimaging tries to say something about a single patient rather than a group. The idea is to chart how healthy brains grow (for example, regional volume as a function of age and sex), then report how far an individual sits from that chart as a Z-score. A Z-score of −3 for the thalamus reads like a clinical finding about that infant. Every such score, though, starts from a regional volume, and that volume depends on where someone, or something trained on someone's labels, drew a boundary. When several experts outline the same developing-brain structure, they disagree substantially. This proposal asks how many individual "extreme deviation" calls would flip if a different equally qualified expert's boundary convention had defined the anatomy. It then tests whether carrying that disagreement into the score, separately for each subject, makes individual deviations more reliable and more clinically informative.
The core idea fits in two sentences. Train a segmentation model that can reproduce each expert's boundary style, push its samples through the normative chart, and get, for each infant, a distribution over Z-scores instead of a single number. Then check whether this subject-specific observer uncertainty changes which infants are flagged as atypical, and whether a cheaper fix, such as inflating every Z-score's denominator by one fixed amount per structure, already does the same job.
Core Claim
Observer disagreement is not a constant measurement noise that can be absorbed into the normative model's variance. It is larger in exactly the brains that normative modeling is meant to flag, such as preterm infants with enlarged ventricles, lower-quality scans, and atypical tissue contrast. It is also correlated across neighbouring structures, because the boundary that one rater assigns to cortical grey matter, another assigns to cerebrospinal fluid (CSF). If this is right, deterministic Z-scores will overstate the confidence of some individual calls and understate others in ways a single per-structure correction cannot repair. A subject-specific treatment should therefore yield deviation calls that are more stable across raters and at least as predictive of outcomes. If it is wrong, and a fixed per-structure inflation matches the full approach, that result is still useful: it would give the field a cheap correction table and show that the expensive rater-modeling machinery adds nothing at the individual level.
Concretely, for subject $i$ with covariates $x_i$ (age at scan, sex), let $\mu(x_i)$ and $\sigma_n^2(x_i)$ be the normative mean and variance of a regional volume. Let $v_i^{(z)}$ be the volume obtained from a segmentation sampled with rater-style latent $z$. The rater-aware deviation is the distribution of
$$Z_i^{(z)} = \frac{v_i^{(z)} - \mu(x_i)}{\sigma_n(x_i)}, \quad z \sim p(z),$$
and the quantity reported clinically is the probability of an extreme deviation, $P_i = \Pr_z(|Z_i^{(z)}| > 2)$, rather than a single thresholded Z. The fixed-inflation alternative replaces this with $Z_i = (\bar v_i - \mu(x_i))/\sqrt{\sigma_n^2(x_i) + \tau_s^2}$, where $\bar v_i$ is the consensus volume and $\tau_s^2$ is a single observer variance for structure $s$ estimated from repeated annotations. The experiments are built to tell these two apart.
Where This Sits
The motivation comes from two literatures that do not cite each other. On the individual-inference side, the preterm normative modeling study fits Gaussian process regression charts for 14 structures on 274 term dHCP infants. It finds that 29% of dHCP preterm infants show extreme positive deviations in ventricular and CSF volume, and that individual Z-scores correlate with ventilation, parenteral nutrition, and Bayley-III outcomes. Its volumes, however, come from one deterministic pipeline. It could not separate the systematic shifts between dHCP (0.5 mm) and EPrime (1.0 mm, different scanner) from true biological differences. Our proposal takes its normative model and cohorts as the testbed and adds the missing measurement layer. More broadly, the legibility-limits analysis argues that individual-level prediction from images plateaus, and the ABCD dynamic connectivity study shows that small effect sizes undermine single-subject use. Neither asks how much of the residual comes from how the anatomy was labeled. We do not claim observer variance explains those plateaus; we measure what fraction of individual-level unreliability it accounts for in one well-defined setting.
On the segmentation side, three methods supply the machinery. Pionono gives each rater a Gaussian in an 8-dimensional latent space, separating systematic bias (the mean) from self-inconsistency (the variance), and reports Cohen's κ = 0.758 on Gleason 2019. The Latent Doctor Model uses a conditional variational autoencoder with a set encoder over a rater's example annotations, and its predictive entropy tracks panel disagreement (Spearman up to 0.728 on ISUP). The variational inference method of Liu et al. generates multi-rater boundaries on QUBIQ brain-growth MRI with continuous Dice 0.5372, against 0.5083 for Bayes dropout. All three stop at image-space agreement. We reuse their rater-latent design and judge it by a criterion they never test: whether it changes individual clinical deviation calls.
The closest connecting work is DiFF. It reweights multi-rater masks so that the fused boundary best serves a downstream diagnostic classifier, and reports AUC gains of up to 2.73% on REFUGE-2. DiFF collapses disagreement into one task-optimized mask. We keep the disagreement as a distribution and target a continuous individual deviation score rather than a class label. The two approaches embody the tension the file identifies: is rater disagreement noise to be optimized away, or information to keep? Our experiments include a DiFF-style collapse so that this tension is tested directly rather than argued.
The FeTA 2024 challenge constrains expectations. Its best automated fetal biometry reached 7.72% mean absolute percentage error, against 9.56% for a gestational-age-only baseline and 5.38% between human raters. Human disagreement is therefore of the same order as the gains automated methods offer, which is exactly why it cannot be ignored when scoring individuals. FeTA also reports Dice drops of about 0.10 with image quality, which supports our hypothesis that observer uncertainty is heteroscedastic, meaning it varies from scan to scan. QUBIQ adds the warning that ensembles of plain deterministic models often beat probabilistic architectures at matching multi-rater uncertainty. That finding shapes our baselines.
Research Program
The first component is data. No dataset in the file has multiple expert labels on the dHCP or EPrime cohorts, so we will collect them at limited scale. Three or four neonatal neuroanatomy raters will each independently correct automated segmentations of the structures whose boundaries are most contested: cortical grey matter, white matter, lateral ventricles, extracerebral CSF, and thalamus. The target is roughly 60 term and 60 preterm dHCP scans plus 40 EPrime scans. Each rater will re-annotate a subset of about 15 scans weeks later to estimate intra-rater variability. To keep cost manageable, raters correct every few slices and the corrections are interpolated; we will report how this protocol itself affects volume agreement. The QUBIQ brain-growth task, with seven experts, serves as an external pretraining and validation set for the rater model.
The second component is a 3D rater-conditioned segmenter. It is a nnU-Net-style backbone with a Pionono-style per-rater latent injected near the output, trained on the corrected labels. Because full-volume latent sampling is the memory bottleneck QUBIQ highlights, sampling happens only in a light final head over shared backbone features, so drawing 50 rater samples costs one backbone pass. With only a few raters, the learned per-rater distributions are thin; the latent's job is to reproduce known styles and plausible mixtures, not to invent new ones. We will check this by holding out one rater and asking whether that rater's volumes fall within the sampled range.
The third component is the rater-aware normative model. We refit the Gaussian process charts from the preterm study on term-infant volumes sampled across rater styles, so the reference chart also reflects boundary convention. We then compute each preterm infant's distribution of Z-scores and the probability $P_i$ of an extreme deviation. Applying the identical pipeline to EPrime lets us split the dHCP-to-EPrime shift into a part that moves with rater style and a part that does not. That split is a direct, if partial, answer to the confound the original authors could not resolve.
Baselines
The naive reference point is the published practice: deterministic Gaussian process Z-scores computed from a single automated segmentation, as in the preterm normative study. It reports the 29% extreme-ventricle rate and significant clinical correlations, but no reliability or rater-flip figure. For the stability of individual calls under annotator change, there is no reported result found on Lacuna. We rerun it on the same subjects with the same covariates and the same hyperparameter search, so any difference comes from the measurement layer alone.
The strongest simple competitor is the fixed per-structure inflation described in the Core Claim. It uses consensus volumes, which means STAPLE (a standard algorithm that fuses several annotators' masks into one estimated true mask), plus one observer variance $\tau_s^2$ per structure estimated from the multi-rater subset. It is strong because it captures most of what observer noise does on average at almost no cost. It is the ablation that isolates our key ingredient: subject-specific, cross-structure-correlated uncertainty. It gets exactly the same annotations. Beating it means a lower rate of flipped extreme-deviation calls on held-out raters and better calibrated coverage of human-rater Z spread, and specifically doing so in the low-quality-scan and high-deviation strata where we predict the gap. A tie overall with a win only in those strata is still informative and will be reported as such.
The strongest existing uncertainty method is a per-rater deep ensemble: one nnU-Net trained per rater, which is affordable with three or four raters. QUBIQ found ensembles to be the most reliable way to capture multi-rater disagreement. Top-team Dice was 0.80–0.85, but the file reports no figure for downstream volumes or Z-scores, so the benchmark there is our own rerun. On QUBIQ brain growth, the comparison point for image-space quality is the 0.5372 continuous Dice of Liu et al.. Ensemble and latent model share the same training budget in GPU-hours and the same augmentation. We count the latent model as worthwhile only if it matches the ensemble on Z-score coverage while sampling more cheaply. If the ensemble wins, the ensemble becomes the recommended method and the downstream conclusions stand.
We also include Monte Carlo dropout, which keeps random dropout active at test time to produce varied segmentations, and a DiFF-style collapse, in which raters are reweighted to maximize prediction of Bayley-III outcome from regional volumes. Both are trained and tuned on the same folds.
Why not just…?
Why not just average the raters with STAPLE and use the consensus volume? Consensus removes rater variability from the number but not from the truth. The question here is whether a single infant's flag is robust to boundary convention, and a consensus volume cannot answer that by construction. The Latent Doctor Model already shows that majority-vote training yields overconfident outputs on borderline cases. The decisive test is our held-out-rater flip rate: if consensus Z-scores rarely flip when a held-out expert's segmentation replaces the consensus, this objection wins and we will say so.
Why not just inflate the normative variance by a fixed observer term per structure? This is the real threat and the reason it is the central ablation. If observer variance is roughly the same across infants, fixed inflation is optimal and our machinery is overhead. Our hypothesis is falsifiable here. We will regress per-subject human-rater volume spread on image-quality scores, preterm status, and deviation magnitude. A flat relationship would mean fixed inflation suffices, and the contribution would shrink to the measurement study plus the correction table.
Why not just use Monte Carlo dropout or test-time augmentation? These capture model uncertainty under input perturbation, not structured differences in where experts put a boundary. On QUBIQ brain growth, Bayes dropout reached 0.5083 continuous Dice against 0.5372 for a rater-latent model. Whether that image-space gap matters for Z-scores is unknown, so dropout is run as a full downstream baseline rather than dismissed.
Why not skip segmentation altogether, as the optimal transport index does? The optimal transport index computes a single Wasserstein distance to a template and beats hippocampal volume for Alzheimer's classification on ADNI (82.30% vs 80.16%). It yields one global number, however, while the clinical value of neonatal normative charts lies in which structure deviates, for example ventricles versus thalamus. The lesion-symptom mapping study likewise found that global lesion volume explained almost nothing (R² = 0.013) compared with spatial models. Segmentation-free indices remain a useful sanity comparison for overall preterm-versus-term separation, but they do not replace regional scores.
Proposed Experiments
The first decisive check is the flip rate, run as leave-one-rater-out. For each preterm infant and structure, we compute deterministic Z from the held-out rater's own segmentation and compare its extreme-deviation status with each method's prediction. The metrics are the fraction of calls that flip and the calibration of $P_i$: among infants with $P_i \approx 0.7$, about 70% should be flagged under the held-out rater. A substantial flip rate in ventricles and CSF would by itself establish that current individual normative reports overstate their certainty.
The second check is clinical. For each method, we test whether ventilation duration, parenteral nutrition, and Bayley-III cognitive, language, and motor scores are better predicted or more strongly correlated when infants are weighted by $P_i$ instead of thresholded deterministic Z. Cross-validation is grouped by subject, with confidence intervals by bootstrap. Better stability with no clinical gain would still matter for reporting, but we would state plainly that it does not improve outcome prediction. The DiFF-style collapse is the comparison that tells us whether keeping disagreement beats optimizing it away.
The third check is cohort transfer. Fitting on dHCP and applying to EPrime, we report how much of the mean regional shift lies within the range of rater-style variation and how much lies outside it. This turns the original study's unresolved confound into an estimate with an interval.
Finally, as a replication on a public multi-rater benchmark, we apply the same pipeline to FeTA fetal biometry. Rater-aware intervals on predicted measurements should cover the human inter-rater spread, which corresponds to 5.38% error between raters. Showing that the best automated model's 7.72% error decomposes into an observer-attributable part and a residual part would show whether the method's value carries over beyond the neonatal cohorts where we collected labels.
References
- Phenotyping the Preterm Brain: Characterizing Individual Deviations From Normative Volumetric Development in Two Large Infant Cohorts
- Computational limits to the legibility of the imaged human brain
- Cognitive and psychiatric relevance of dynamic functional connectivity states in a large (N > 10,000) children population
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Advances in automated fetal brain MRI segmentation and biometry: Insights from the FeTA 2024 challenge
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
- An Optimal Transportation Based Univariate Neuroimaging Index
- Individualized lesion-symptom mapping using explainable artificial intelligence for the cognitive impact of white matter hyperintensities