When two radiologists outline the same lung nodule, their outlines differ, and the difference can move a nodule's measured size across a management threshold. That can decide whether a patient gets a follow-up scan in a year or a biopsy next week. Health economists already have a standard way to ask whether decisions hold up under uncertainty. Probabilistic sensitivity analysis (PSA) draws every uncertain model input thousands of times from a probability distribution, reruns the cost-and-outcome model on each draw, and reports how often a strategy remains worth its price. In imaging-dependent decisions, though, PSA summarizes the image-reading step as one scalar diagnostic accuracy with a Beta distribution, as in the opportunistic CT osteoporosis model of Rühling et al.. Meanwhile, multi-rater segmentation models learn full distributions over plausible outlines but stop at geometric scores.
This proposal feeds case-level samples from a multi-rater segmentation model directly into the PSA Monte Carlo loop. The central methodological claim is that the two kinds of reader disagreement belong in two different places in a health-economic model. Case ambiguity, meaning that this particular nodule is genuinely hard to outline, is patient-level variability. Because management rules are thresholds, it shifts the expected net benefit of a strategy. Systematic rater bias, meaning that this reader tends to draw large, is a shared effect across every patient that reader sees. It belongs in the second-order (population-level) uncertainty that widens acceptability curves and drives value-of-information estimates. Collapsing both into one Beta on sensitivity gets both effects wrong. The proposal also yields a concrete decision tool: a per-case estimate of what a second reader is worth, in currency.
Core Claim
Consider a size-threshold decision rule in which a scalar measurement $m$ (nodule diameter or volume) derived from a contour selects a management action. For such a rule, the decision-relevant quantity is how segmentation uncertainty is distributed relative to the threshold, case by case. The average amount of disagreement is not the relevant quantity. Hard-to-outline nodules are not spread uniformly across sizes. If ambiguity concentrates near thresholds, a case-independent accuracy parameter will misstate the expected net benefit of a management strategy and the value of resolving the ambiguity.
We make two testable predictions. First, case-level propagation changes the mean net monetary benefit relative to scalar-accuracy PSA, through the threshold nonlinearity. Second, only the rater-bias component, modeled as a shared random effect, materially widens the cost-effectiveness acceptability curve (the probability of cost-effectiveness plotted against willingness to pay) and raises the expected value of perfect information. If neither prediction holds on real multi-rater data, the scalar shortcut is adequate. That is a publishable and useful negative result.
Relation to Prior Work
The segmentation side supplies the method. The Probabilistic U-Net samples whole, spatially coherent alternative outlines from a low-dimensional latent code. Pionono goes further and gives each rater a Gaussian in latent space: its mean captures that rater's systematic bias and its variance captures that rater's inconsistency. The Latent Doctor Model learns annotator style from a small set of that doctor's labels. Mixture of Stochastic Experts matches the full, multimodal distribution of annotations with an optimal-transport loss.
All four evaluate only with distributional or geometric scores such as generalized energy distance, Cohen's kappa, and calibration error. None asks whether their samples change a clinical decision. Pionono's split of rater effects into mean and variance is exactly the structure we need, because it maps onto the PSA distinction between shared and patient-level uncertainty. We reuse it rather than invent a new architecture.
The closest work connecting segmentation disagreement to downstream use is the diagnosis-first calibration of Wu et al.. They weight raters by how much their labels help a downstream classifier and report diagnostic AUC gains of up to 2.73% on REFUGE-2. That work motivates ours: it shows that rater "quality" depends on the downstream task. It differs from ours in kind. It optimizes a training label for classification accuracy, whereas we keep the full disagreement distribution and price it in costs and quality-adjusted life years (QALYs) under a willingness-to-pay threshold.
The health-economics side supplies the evaluation machinery. The oQCT Markov model of Rühling et al. runs 30,000 Monte Carlo iterations but treats CT measurement accuracy as fixed literature values. The AmbIGeM analysis supplies the value-of-information template (expected value of perfect information and expected net benefit of sampling) that we adapt to the purchase of expert annotations. The radiotherapy contouring literature, such as the autosegmentation review and SegTHOR, argues that contour variation matters clinically but treats it as a problem to be edited away, not as an input to a decision model. We deliberately avoid radiotherapy as the primary testbed: dose–toxicity modeling would add a second uncertain model that could swamp the effect we want to isolate.
Research Program
The primary testbed is LIDC-IDRI, which provides thoracic CT nodules each outlined independently by up to four radiologists. The decision model is a decision tree over nodule management. A size-threshold rule, of the kind used in lung screening guidelines, maps a nodule's measured diameter or volume to routine surveillance, short-interval CT, or tissue sampling. Downstream branches carry malignancy probability as a function of true size, procedure complication rates, costs and utilities. These enter as ordinary PSA distributions (Beta for probabilities and utilities, Gamma for costs) with parameters from the published literature.
An honest limitation: LIDC lacks pathology truth for most nodules. The "true" size is therefore represented by the rater population itself, and malignancy risk comes from external literature. The study isolates the measurement-to-decision link, not LIDC's own outcomes.
The propagation works as a nested Monte Carlo. In each outer iteration we draw the usual economic parameters and, separately, one rater-bias vector per simulated reading program. This bias vector is the Pionono-style rater mean, drawn from the learned distribution over raters and held fixed across all patients in that iteration. In the inner loop, for each case in the cohort, we sample a contour conditioned on that rater, compute $m$, apply the rule, and evaluate costs and QALYs. The key ingredient is holding the rater bias fixed across patients within an iteration; the ablations remove exactly this.
The value of a second read for case $i$ is computed from the net monetary benefit $NB(d, m) = \lambda E(d, m) - C(d, m)$. Here $d$ is the management action, $\lambda$ is willingness to pay per QALY, and $E$ and $C$ are expected QALYs and costs given the true measurement $m$. The perfect-information value of resolving that case's contour is
$$V_i = \mathbb{E}_{m \sim p_i}\left[\max_d NB(d, m)\right] - \max_d \mathbb{E}_{m \sim p_i}\left[NB(d, m)\right],$$
where $p_i$ is the model's distribution over the case's measurement. A single extra reader is partial information. For that we compute a sample-information analogue by conditioning $p_i$ on one additional sampled rater's contour. A policy that requests a second reader whenever this per-case value exceeds the reader's cost is the practical output.
The secondary testbed checks portability. We port the same decomposition to Gleason 2019, where six pathologists' grading disagreement maps onto a surveillance-versus-treatment decision. There Pionono's rater structure already exists, so the only new piece is the decision model.
Baselines
The naive reference measures each nodule on a consensus mask and runs the decision model with point-estimate accuracy and no reader uncertainty. Any sound method should differ from it only when disagreement straddles thresholds.
The strongest simple competitor is conventional PSA with inflated scalar accuracy. We estimate the rate at which raters disagree on management category from the LIDC training cases, fit a Beta distribution to that rate, and sample it in the outer loop exactly as Rühling et al. handle accuracy. This is how a competent health economist would do it today. It uses the same data, the same decision tree and the same number of iterations as our method. No decision-level result for it on LIDC-IDRI exists: there is no reported result found on Lacuna for decision-level or economic metrics on LIDC-IDRI. For calibration of expectations, the closest reported economic robustness figure is oQCT being cost-effective in 88–90% of PSA iterations (Rühling et al.).
The reference target is empirical rater resampling. On held-out LIDC cases we draw directly from the actual four radiologists' contours, bootstrapping over raters so that one rater is shared across the cohort in each iteration. This is the ground truth that any learned sampler must reproduce. A learned model has no advantage here when real multiple reads exist; its purpose is to supply the distribution when only one scan and no extra reads are available.
The competing learned samplers are Monte Carlo dropout and a five-member deep ensemble of U-Nets, alongside the Probabilistic U-Net and Pionono. All are trained on the same splits with matched tuning budgets. For segmentation quality we report generalized energy distance. The file gives no numerical LIDC value for the Probabilistic U-Net or MoSE, only statements of significant improvement. The strongest reported rater-modeling numbers are Pionono's kappa of 0.758 on Gleason 2019 and the Latent Doctor Model's KL divergence of 0.461 there.
The ablation isolating our key ingredient runs Pionono samples in two ways. In one, rater bias is pooled into each case's variance, so each case draws an independent rater. In the other, rater bias is shared across the cohort within an iteration. Beating the scalar baseline means the case-level method lands closer to the empirical-rater reference than scalar inflation does. We measure this by the maximum absolute gap in the acceptability curve across willingness-to-pay values and by relative error in expected value of perfect information, with a pre-specified gap of more than 0.05 on the curve or more than 10% on that value.
Why not just…?
Why not just inflate the Beta on diagnostic accuracy using observed inter-rater agreement? This is the real threat, and the experiment is designed to settle it. It fails in principle only if disagreement is concentrated near thresholds, and only if rater bias is correlated across patients. The first creates a mean shift that a case-independent parameter cannot represent. The second creates population-level uncertainty that per-patient noise averages away. If on LIDC the scalar baseline matches the empirical-rater reference within our margins, we will report that the shortcut is adequate for size-threshold decisions. We will then redirect the method to the per-case second-read policy, where a scalar has nothing to say by construction.
Why not just resample the actual raters and skip the generative model? Where multiple real reads exist, one should, and we use this as the reference. It is unavailable at deployment, where a scan has one reader or one AI output. It also cannot extrapolate beyond the few raters in a dataset. The learned sampler earns its place only if it reproduces the empirical-rater decision distribution on held-out cases. That comparison is a headline result, not an assumption.
Why not just use Monte Carlo dropout or an ensemble on a standard U-Net? The Probabilistic U-Net paper reports that dropout yields incoherent pixel-wise noise and that ensembles miss rare modes on LIDC. Neither separates rater bias from case ambiguity, and that separation is the mechanism under test. Whether this matters for decisions, rather than for energy distance, is unknown. So we run both as samplers and compare their decision-level error against the empirical-rater reference.
Why not just add a fixed safety margin or use consensus labels? A fixed margin corresponds to shifting the measurement threshold. We will include it as a policy arm, since in the decision model it is just another action. Consensus labels discard exactly the variance at issue, as the Latent Doctor Model and Pionono document. That is why consensus measurement serves as the naive reference.
Decisive Checks
Three results decide the proposal.
The first is whether case-level propagation moves expected net benefit and the acceptability curve beyond the pre-registered margins relative to scalar inflation, with the empirical-rater reference as arbiter.
The second is the ablation. Sharing rater bias across the cohort should widen the acceptability curve and raise the expected value of perfect information, while pooled bias should not. If both variants give the same curves, the first-order/second-order mapping is wrong for this setting.
The third is whether a learned sampler reproduces held-out empirical-rater decision distributions better than dropout or ensembles. If it does, the targeted second-read policy should then deliver higher expected net benefit than both universal double reading and no double reading at realistic reader costs.
We will count the proposal as successful if the first and third checks pass. If the first check fails, we will report that the scalar shortcut suffices, with the Gleason 2019 port testing whether that conclusion generalizes.
References
- Cost-effectiveness of opportunistic QCT-based osteoporosis screening for the prediction of incident vertebral fractures
- A Probabilistic U-Net for Segmentation of Ambiguous Images
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Cost-Effectiveness and Value of Information Analysis of an Ambient Intelligent Geriatric Management (AmbIGeM) System Compared to Usual Care to Prevent Falls in Older People in Hospitals
- Deep learning for autosegmentation for radiotherapy treatment planning: State-of-the-art and novel perspectives
- Segthor: Segmentation of Thoracic Organs at Risk in CT Images