Grading Segmentation Uncertainty by the Clinical Outcomes It Changes

When several expert radiologists outline the same lung nodule on a CT scan, they draw different boundaries. A family of "probabilistic segmentation" models learns to reproduce this disagreement by producing many plausible outlines per image instead of one. These models are scored almost entirely with geometric measures of how well their outlines overlap the experts' outlines. A clinic, however, does not use an outline for its shape. It uses it to decide whether to treat, to aim a radiation beam, and to judge how much healthy lung gets irradiated. Those downstream quantities are uneven and asymmetric: a boundary that strays one millimetre toward healthy lung costs more than one that strays into air. So two models with identical overlap scores can disagree sharply on what they imply for the patient.

This note proposes to score uncertainty models in outcome space rather than pixel space. Each expert outline, and each model-sampled outline, is passed through a simplified but explicit clinical pipeline: a malignancy call, a radiotherapy plan, a tumor-coverage estimate, and a lung-toxicity estimate. The question is whether the model reproduces the spread of outcomes that real expert disagreement produces. The central empirical bet is falsifiable. Rankings of uncertainty methods by geometric distance will disagree with rankings by outcome-distribution distance. If a cheap geometric metric weighted by distance to nearby organs reproduces the outcome ranking, the outcome machinery is unnecessary, and we will say so.

Core Claim

The "comprehensive outcome assessment" tradition in clinical research judges an intervention jointly on efficacy, toxicity, patient-reported burden and cost, because a single endpoint misleads. Examples include a meta-analysis weighing survival against quality of life and adverse events for docetaxel in prostate cancer (Can docetaxel combined prednisone effectively treat hormone refractory prostate cancer?) and a meta-analysis comparing robotic and laparoscopic pancreatic surgery across a dozen complication endpoints (Short-term comparison of Robotic and Laparoscopic pancreatoduodenectomy). These studies motivate the proposal, but they never ask how imaging delineation choices feed those endpoints.

Inter-observer segmentation work has the opposite blind spot. The Probabilistic U-Net (A Probabilistic U-Net for Segmentation of Ambiguous Images) uses a learned low-dimensional random code so that each sample is a coherent whole outline. Its hierarchical successor (A Hierarchical Probabilistic U-Net for Modeling Multi-Scale Ambiguities) uses a coarse-to-fine stack of such codes. OMBA-UNet (Inter-Rater Uncertainty Quantification in Medical Image Segmentation via Rater-Specific Bayesian Neural Networks) gives each rater a separate decoder. All three are judged geometrically.

Our claim is that the right test of such a model is joint: does it reproduce the multi-endpoint outcome distribution that the real panel of raters induces? The joint test matters because the endpoints trade off against each other. A larger outline improves tumor coverage and worsens lung dose. A model that only gets the marginal spread of each endpoint right can still get the trade-off wrong.

Relation to the Closest Work

Two papers come closest, and the proposal is defined against them.

DiFF (Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle) learns how to weight raters so that the fused outline maximizes a downstream diagnosis network's accuracy. It reports diagnostic AUC gains of up to 2.73% on REFUGE-2 glaucoma, 2.51% on TNMIX thyroid ultrasound, and 2.04% on ISIC melanoma. DiFF supplies the idea that a downstream task should define what a "good" outline is. It differs from our proposal in three ways. It uses one binary endpoint. Its downstream model is a learned classifier rather than an explicit treatment model. And it outputs a single fused outline rather than evaluating a distribution.

SOARS (Comprehensive and clinically accurate head and neck cancer organs-at-risk delineation on a multi-institutional study) automates 42 head-and-neck organ outlines. It shows that the dose deviations its outlines cause are within or smaller than human inter-user variation. It is the closest to propagating outlines into physics. It is deterministic, however. It compares one AI plan against human spread rather than asking whether a model's own spread matches the human spread. It also stops at dose, explicitly deferring patient outcomes to future trials.

Two further papers constrain the design. The Zepf et al. analysis (Navigating Uncertainty in Medical Image Segmentation) shows two things. On low-variability prostate data, plain softmax entropy from a deterministic U-Net matches complex models, which overestimate variance. And the standard Generalized Energy Distance (GED, an average pairwise distance between sets of outlines) mixes two kinds of disagreement: whether an object is present at all ("detection") and where its edge lies ("boundary"). We adopt their split and predict that it maps onto endpoints: detection ambiguity should drive the diagnosis endpoint, and boundary ambiguity should drive coverage and toxicity.

The LIDC-IDRI relabeling study (Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction) shows that radiologist malignancy scores carry a defensive over-calling bias. It reports sensitivity 0.97 but specificity 0.23, against 0.67 and 0.66 when labels are anchored to biopsy-referenced cases. This is a warning that rater spread is not pure biological ambiguity. It also gives us an outcome-grounded label to test against.

Research Program

The testbed is LIDC-IDRI. It is the one dataset in the file with four independent radiologist outlines per nodule, rater malignancy scores, and a biopsy-anchored relabeling. Small solitary nodules are also a realistic setting for stereotactic radiotherapy, which delivers a few high doses to a small target. The surrounding lung can be outlined reliably with a fixed deterministic model, so all variation in the pipeline comes from the target outline.

For every nodule we build an outcome pipeline with four endpoints. The diagnosis endpoint is a malignancy probability from a fixed classifier fed the image crop and the outline, scored against the biopsy-anchored labels. For treatment, we expand each outline by a fixed planning margin and generate a dose distribution with a deliberately simple analytic dose model: prescription dose inside the expanded region, falling off with a fixed penumbra. This is not a clinical-grade dose calculation, and we will not present it as one. Its purpose is to make the efficacy–toxicity trade-off explicit and reproducible.

From the dose we compute two quantities. Efficacy is the fraction of the tumor that receives the prescription dose. Toxicity is the mean lung dose converted to a pneumonitis risk through a standard sigmoid dose–response curve, with fixed published-style parameters held constant across all methods. A fourth, operational endpoint is the treated volume, a proxy for delivery burden.

The key design choice is cross-rater evaluation. We plan on rater $i$'s outline and score coverage against rater $j$'s outline. For each nodule this produces an outcome matrix that answers: "if this rater planned and that rater was right, what happens?" A model is evaluated the same way. We plan on each of its samples and score against each real rater. The model's distribution of outcome vectors is then compared with the raters' distribution. We use an energy distance computed over the standardized four-endpoint vector rather than over pixels:

$$ D_{\text{out}} = 2\,\mathbb{E}\lVert y_m - y_r\rVert - \mathbb{E}\lVert y_m - y_m'\rVert - \mathbb{E}\lVert y_r - y_r'\rVert , $$

Here $y_m, y_m'$ are independent outcome vectors from model samples, and $y_r, y_r'$ are independent outcome vectors from rater outlines. This is the same functional form as GED, so the only change from the standard metric is the space in which distance is measured. We also report each endpoint's spread on its own and the coverage–toxicity correlation, to see which part of the joint distribution a model misses.

The secondary analysis tests the bias concern raised by the relabeling study. We ask whether raters who over-call malignancy also draw systematically larger outlines. If so, a model that faithfully copies rater spread is copying a defensive bias into the dose. We would then also report $D_{\text{out}}$ restricted to biopsy-anchored nodules, where diagnosis can be scored against verified truth.

If outcome-space and geometric rankings diverge, a final contingent step fine-tunes the Probabilistic U-Net with an added differentiable outcome term. The coverage and lung-dose surrogates are smooth functions of the soft outline, which makes this possible. We would test whether outcome fidelity improves without loss of GED.

Baselines

The naive reference is a deterministic U-Net trained on the four raters' majority-vote outline. Its only spread comes from planning on that single outline and scoring against each rater. This is what most clinics deploy. A useful method must beat it on $D_{\text{out}}$, and failing to do so would mean modeling disagreement has no outcome value on this data.

The second baseline is softmax-entropy sampling from the same deterministic U-Net, implemented by thresholding its probability map at several levels to make multiple outlines. It is included because Zepf et al. found exactly this kind of cheap estimate competitive on low-variability data, so it is the most credible threat to the claim that structured models are needed.

The third is a deep ensemble of five U-Nets with the same architecture and total training budget per member. The QUBIQ challenge summary (QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge) reports that simple ensembles often beat probabilistic architectures.

The strongest existing method is the Hierarchical Probabilistic U-Net. On LIDC-IDRI it reports reconstruction IoU of 0.97 versus 0.75 for the standard Probabilistic U-Net (A Hierarchical Probabilistic U-Net for Modeling Multi-Scale Ambiguities). We also run the standard Probabilistic U-Net and OMBA-UNet. OMBA-UNet is the natural model if rater-specific bias matters, since it keeps one decoder per rater.

DiFF is included as the task-driven competitor. Its single fused outline is evaluated in the same pipeline. We expect it to help the diagnosis endpoint and possibly hurt coverage, but it is a real comparator on the diagnosis axis, where its reported gains are up to 2.73% AUC on other datasets. No reported result was found on Lacuna for any method evaluated by an outcome-space distance on LIDC-IDRI, so there is no published number to beat on $D_{\text{out}}$ itself.

For fairness, every model uses the same splits, a shared encoder backbone, the same number of samples per nodule (sixteen), and the same hyperparameter search budget tuned on validation GED. That last choice deliberately favours the geometric view. All models feed the identical frozen pipeline.

The ablation that isolates the key ingredient is a proximity-weighted geometric distance. It is GED computed with boundary errors weighted by their signed distance toward the lung, which approximates the main outcome asymmetry without any dose model. Beating standard GED is not enough. The outcome metric earns its place only if it ranks methods differently from this weighted geometric proxy, measured as a Kendall rank correlation clearly below what bootstrap resampling of nodules produces between two runs of the same metric.

Why not just…?

Why not just compute dose on each rater's outline, as SOARS did, and skip the generative models? That measures how much human disagreement matters, and our rater-only cross-planning matrix does exactly that as a by-product. But it requires four expert outlines for every new patient, which is the bottleneck these models exist to remove. The open question is whether a model can stand in for the panel at deployment, and that can only be answered by comparing model-induced and panel-induced outcome distributions.

Why not just add a safety margin around a single outline and let the margin absorb disagreement? Margins are the clinical default, and this is a serious alternative. A fixed margin, however, is the same in every direction and for every patient, whereas disagreement is concentrated on specific ambiguous edges. The experiment settles it directly. We include a deterministic outline with a margin enlarged until its average coverage matches the panel's. If this matches the probabilistic models on $D_{\text{out}}$, including the coverage–toxicity correlation, then per-patient uncertainty modeling adds nothing for planning on this task.

Why not just weight GED by distance to the organ at risk? This is the strongest threat, which is why it is the central ablation rather than a footnote. If it reproduces the outcome ranking, we will report that the cheap proxy suffices and recommend it. The proxy cannot capture the diagnosis endpoint, though, and it cannot capture coupling between endpoints. Our prediction is therefore that it agrees on toxicity alone but diverges on the joint metric.

Why not just use softmax entropy, since Zepf et al. found it competitive? Their finding was for prostate outlines where disagreement is mostly drawing noise. Lung nodules have substantial detection ambiguity, in that raters disagree about whether a nodule is there at all, and thresholded probability maps cannot represent "present in some samples, absent in others" coherently. If entropy sampling nonetheless matches the Hierarchical Probabilistic U-Net on $D_{\text{out}}$, that extends their negative result to outcome space and is worth reporting.

Decisive Checks

The study succeeds or fails on four measurements, all run on the held-out LIDC-IDRI test split with nodule-level bootstrap intervals.

First, the rank correlation between the GED ranking and the $D_{\text{out}}$ ranking across the seven methods, and separately between the proximity-weighted proxy and $D_{\text{out}}$. A low correlation against standard GED but a high one against the proxy means the simple fix is enough.

Second, whether the Hierarchical Probabilistic U-Net's geometric advantage, 0.97 versus 0.75 reconstruction IoU, survives in outcome space. If it shrinks to within bootstrap noise of the ensemble, geometric fidelity is being bought on edges that do not affect outcomes.

Third, whether Zepf's split holds endpoint by endpoint, with detection-term error predicting diagnosis-endpoint error and boundary-term error predicting coverage and toxicity error.

Fourth, whether rater outline volume correlates with rater malignancy over-calling. If it does, then reproducing the panel distribution means reproducing the panel's defensive bias, and the right target shifts to outcome fidelity on biopsy-anchored nodules. That would be the most consequential result for how inter-observer data should be used in treatment planning.