How Much Do Toxicity Predictions Depend on Who Drew the Organ?

Radiotherapy plans are judged partly by models that predict side effects. These normal tissue complication probability (NTCP) models take the dose a patient's healthy organs will receive and return the chance of a complication such as a permanently dry mouth, painful mucositis, or swallowing damage. The worst of these side effects force doctors to pause treatment, and a pause lets the tumor regrow.

The newest NTCP models do much better than the old mean-dose rules because they use 3D organ shape and the spatial pattern of dose. Both kinds of input are computed from an organ outline drawn by one clinician. Different clinicians draw different outlines, and the same clinician draws a different one on another day. Nobody has measured how much a patient's predicted risk, or the clinical decision that follows from it, changes when a different but equally plausible expert draws the organ.

This note proposes that measurement and a fix for what it finds. The plan has three parts. First, collect real repeat outlines of the parotid glands on a head-and-neck cohort that has toxicity follow-up. Second, push many plausible outlines through a fixed treatment plan and an NTCP model. Third, report how often a patient's risk category flips. If the flip rate is large, the gain of shape-and-dose-pattern models over mean-dose models is partly a measurement artifact. We then test whether averaging features over the distribution of plausible outlines keeps discrimination while making individual risk estimates stable.

Core Claim

The claim has two parts. The first is that the reported gain of radiomic and dosiomic xerostomia models comes partly from features that are unstable under inter-observer contour variation. Radiomics here means descriptors of organ shape such as volume and eccentricity. Dosiomics means descriptors of the spatial dose pattern, such as directional gradients. Design and Selection of Machine Learning Methods Using Radiomics and Dosiomics for Normal Tissue Complication Probability Modeling of Xerostomia reports AUCs of 0.74–0.88 for these models against below 0.60 for mean parotid dose. Every one of its features was extracted from a single manual contour, and its variance across plausible contours was never measured. The same weakness is documented for radiomic features in other settings (Computed Tomography Radiomic Nomogram for Preoperative Prediction of Extrathyroidal Extension in Papillary Thyroid Carcinoma).

The second part is that the right unit of evaluation is the decision, not the pixel overlap. We will measure the fraction of patients whose predicted risk crosses a planning-relevant threshold under some plausible outline but not under others. This is the "decision flip rate." It turns the "uncertainty is hard to act on" complaint raised by QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge into a single clinically legible number.

The closest conceptual precedent is Calibrate the inter-observer segmentation uncertainty via diagnosis-first principle. It argues that rater disagreement should be judged by its effect on a downstream task, and it shows 2–2.7% AUC gains in 2D diagnosis. We borrow the principle but not the method. Our downstream task is complication prediction on 3D dose. Our goal is to quantify and marginalize over disagreement, not to learn a single "best" outline.

Research Program

Data. The data foundation is a targeted recontouring effort. We would use the Heidelberg head-and-neck cohort of 153 patients with CTCAE toxicity reports at 0–6, 6–15 and 15–24 months, used by the dosiomics paper. On a stratified subset of about 40 patients, enriched for patients with events, four radiation oncologists would each recontour both parotid glands twice, blinded to outcome. Where feasible they would also contour the pharyngeal constrictors and oral cavity.

Why those extra structures. The parotids drive chronic xerostomia. The thin mucosal and muscular structures drive the acute, treatment-limiting toxicities that cause interruptions, and thin structures are exactly where automatic and human contours disagree most. SegTHOR reports esophagus Dice of 0.80 against 0.93 for heart (Segthor: Segmentation of Thoracic Organs at Risk in CT Images). SegRap2023 finds thin structures hardest (SegRap2023: A Benchmark of Organs-at-Risk and Gross Tumor Volume Segmentation for Radiotherapy Planning of Nasopharyngeal Carcinoma). Whether grade ≥3 acute mucositis or dysphagia is recorded densely enough in the 0–6 month reports to support a secondary endpoint must be checked before committing to it.

Generating plausible outlines. We need to generate plausible outlines for the whole cohort, not only the recontoured subset. We will pretrain a segmentation network on SegRap2023's 45-organ annotations. We will then fit a rater-aware variability model to the recontoured subset. For this we use Pionono (Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation), which represents each rater as a Gaussian in a small latent space, so its mean captures rater bias and its variance captures self-inconsistency. This lets us sample both between-rater and within-rater variation. As a competitor we use a deep ensemble. QUBIQ found that ensembles often match more elaborate probabilistic models, and The Latent Doctor Model for Modeling Inter-Observer Variability disputes that finding. We treat the question as empirical rather than taking a side.

Propagation. For each patient we draw about 50 outlines. Each is evaluated on the delivered dose grid of the fixed clinical plan, so no dose prediction network is needed at this stage. We then recompute mean dose and the full radiomic and dosiomic feature set, and pass each set through the retrained NTCP models. This gives, for every patient, a distribution of predicted risk rather than a point.

The fixed-plan choice isolates evaluation sensitivity: the same dose read through a different outline. The additional effect of a different outline producing a different optimized plan is deliberately deferred. It would require a contour-conditioned dose predictor such as Combining distance and anatomical information for deep-learning based dose distribution predictions for nasopharyngeal cancer radiotherapy planning or DiffDP: Radiotherapy Dose Prediction via a Diffusion Model. Both take a single contour as fixed input, so plugging sampled contours into them is the natural extension. Subgroup-Specific Risk-Controlled Dose Estimation in Radiotherapy shows that dose error bounds calibrated globally fail in the high-dose region, which warns that any later contour-to-dose extension must be calibrated on the organ-boundary voxels specifically.

The remedy. The remedy we test is a contour-marginalized NTCP model. It is trained on features averaged over sampled outlines, and at prediction time it reports both the mean risk and its spread. We compare it with the standard radiomics practice of discarding features whose between-outline agreement is low.

Interpretive guardrail. We frame every result as predictive. Considering causality in normal tissue complication probability model development: a literature review found that 43.9% of head-and-neck NTCP papers make causal dose recommendations without any adjustment for confounders. Our flip rate is a statement about the reliability of a prediction, not a claim that changing the dose would change the outcome.

Baselines

Naive reference. The naive reference is the mean-parotid-dose logistic NTCP computed on the single clinical contour. On the same Heidelberg cohort it is reported at AUC below 0.60 (Design and Selection of Machine Learning Methods Using Radiomics and Dosiomics for Normal Tissue Complication Probability Modeling of Xerostomia). Mean dose is expected to be the most contour-robust quantity. We therefore report both its AUC and its flip rate, since it sets the floor for stability.

Strongest existing method. The strongest competitor is that paper's radiomic and dosiomic pipeline, support vector machines and extra-trees on shape and dose-pattern features, reported at AUC 0.74–0.88 for long-term xerostomia. We will rerun it with the paper's model families and a matched hyperparameter search budget, using the same outcome definitions and the same cross-validation folds as our model. Beating it does not mean a higher single-contour AUC. It means cross-validated AUC within 0.02 of it, clearly narrower per-patient risk spread, and a lower flip rate when both are evaluated on outlines sampled from the held-out real recontours. For decision flip rate itself, no reported result was found on Lacuna for any NTCP model.

Ablations isolating the key ingredient. The key ingredient is a realistic model of expert disagreement, so the ablations swap it out. The first replaces the learned rater distribution with a deep ensemble of the same backbone at equal training compute. The second replaces it with synthetic isotropic erosion and dilation of the clinical contour by 1–3 mm. The third uses a feature-stability filter without marginalization. Each variant's predicted flip rate is scored against the flip rate measured from the real four-rater, two-session recontours. This is the only ground truth for "plausible variation" we have, so it serves as the arbiter.

Why not just…?

Why not just add a planning organ-at-risk margin and move on? A uniform expansion margin protects the organ during planning but does nothing to the NTCP evaluation, which is our object of study. A risk prediction read from a margin-expanded organ would be just as contour-dependent. Margins also assume uniform, direction-free uncertainty. That assumption is wrong for steep conformal dose gradients, whose directional components are exactly what the dosiomic features encode.

Why not just use STAPLE or majority vote to get one clean contour? A consensus outline gives one risk number, but it hides whether that number is fragile. The diagnosis-first work shows that majority voting loses downstream accuracy relative to task-aware fusion (Calibrate the inter-observer segmentation uncertainty via diagnosis-first principle). Our experiments include STAPLE-derived features as an arm, so if consensus features are as stable and as predictive as marginalized ones, the data will show it. The practical objection is that consensus requires several raters per patient at deployment, while a learned variability model does not.

Why not just perturb contours with random erosion and dilation instead of learning rater distributions? This is the real threat. It is cheap and may capture most of the variance that matters for volume and mean dose. It is likely to miss structured disagreement, such as where the deep lobe of the parotid ends, and that structure could dominate shape features like eccentricity. The ablation above settles it directly. If synthetic perturbation reproduces the measured real-rater flip rate within its confidence interval, we will recommend it, and the learned model becomes unnecessary for this purpose.

Why not just drop unstable features using an intraclass-correlation filter? Stability filtering is standard radiomics practice and is included as an arm. It can remove exactly the geometric features that carried the gain over mean dose. The question is whether marginalization keeps predictive signal that filtering throws away, and this is measured by the AUC gap between the two at equal stability.

Proposed Experiments

Real-rater measurement. The first experiment uses only the recontoured subset and involves no model. For each patient we compute NTCP under all eight real outlines, which gives the empirical spread of risk and the flip rate at a xerostomia risk threshold fixed in advance. The threshold is taken from the cohort's event-rate calibration, not tuned. This number alone answers the headline question.

Cohort-wide sampled variation. The second experiment extends this to all 153 patients using sampled outlines. We report cross-validated AUC, calibration slope, mean per-patient risk spread, and flip rate for mean-dose, radiomic/dosiomic, stability-filtered, STAPLE, and contour-marginalized models. We also report per-feature between-outline agreement, so that it is visible which features carry the instability.

Validation of the variability generators. The third experiment validates the sampled outlines against the held-out real recontours. We use generalized energy distance and continuous Dice, following QUBIQ practice, and above all the agreement between flip rates computed from sampled versus real outlines.

Decisive outcomes. The decisive result would be twofold. First, the single-contour dosiomic model's advantage over mean dose narrows markedly when evaluated with real rater contours, or its flip rate is substantially higher than mean dose's. Second, the marginalized model restores stability without falling back to mean-dose discrimination. If instead dosiomic features prove stable, with flip rates near those of mean dose, that is a clean negative result. It would remove a standing worry about these models and would be worth publishing as such.

Secondary endpoint. If acute grade ≥3 mucositis or dysphagia counts allow, the same pipeline is repeated for constrictor and oral-cavity contours. The prediction is that flip rates are higher there than for the parotids, because thinner structures should show more contour-driven risk variation.