Where Experts Disagree: Using Survival Outcomes to Choose Among Plausible Tumor Boundaries

When several radiologists outline the same tumor on a scan, they often draw different borders. The disagreement is worst where the tumor fades gradually into healthy tissue rather than ending at a sharp edge. Most pipelines that predict how long a patient will survive from imaging average these borders into one "consensus" mask and then forget the disagreement happened. This proposal tests a different idea. The band of tissue where experts disagree may itself carry prognostic signal, because infiltrative margins are plausibly where aggressive disease lives. Patient outcomes can then be used to decide which expert-plausible border matters for prognosis. Outcomes are never allowed to invent a border no expert would draw.

Concretely, we would learn the range of plausible expert contours from a dataset where every lesion was traced by four radiologists, and transfer that range to lung cancer cohorts that have survival follow-up but no multi-rater masks. A survival model then reads image features inside the agreed-upon core, inside the disputed band, and inside each candidate border. Only a small selection mechanism is trained on outcomes. This addresses the expert annotation bottleneck, because survival cohorts never need repeated expert tracing. It also tests an open tension in the literature about whether boundary precision matters for prognosis at all.

Core Claim

Two claims are tested, and each is separately falsifiable. First, features from the inter-rater ambiguity band add prognostic information beyond the consensus lesion, age, and stage. This is measured as concordance index (C-index: the probability that, of two patients, the one predicted to be at higher risk has the event first) on held-out and external cohorts. Second, outcome information should select among expert-plausible boundaries, not reshape them. A survival loss allowed to freely move a segmentation boundary will degrade anatomical fidelity and generalize worse across cohorts than one restricted to a mixture over expert-like hypotheses.

The second claim is the more delicate one. It may turn out that a fixed, non-learned use of the band does just as well. If so, the honest conclusion is that the disagreement band carries the signal and outcome-driven selection is unnecessary. The experiments are designed so that this outcome is informative rather than a failure.

Relation to Prior Work

The closest existing work is Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle (DiFF). DiFF learns spatial "expertness maps" that weight each rater's mask according to how well the fused mask supports a downstream diagnosis classifier. It reports diagnostic AUC gains of up to 2.73% (glaucoma), 2.51% (thyroid) and 2.04% (melanoma) over multi-rater fusion baselines. DiFF supplies the central intuition that downstream task utility can arbitrate rater disagreement.

We differ in three ways. Our outcome is right-censored time-to-event, not a class label. DiFF requires several complete expert masks for every training patient in the target cohort, whereas we require them only in a separate source dataset. And DiFF learns dense per-pixel weights, whereas we deliberately restrict outcome-driven learning to a low-dimensional choice among expert-like hypotheses. With a few hundred survival patients, a dense weighting would have far more freedom than the outcomes can constrain.

The method for representing expert disagreement comes from Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts (MoSE). MoSE trains several stochastic decoder heads and a gating network so that the spread of predicted masks matches the spread of multi-radiologist annotations on LIDC-IDRI. Our addition is to re-purpose MoSE's gate as the only spatial component trained by the survival objective. MoSE itself treats all rater modes as equally plausible and never asks which mode is clinically relevant.

Three papers motivate the question and also constrain the design.

Post-Hoc Overall Survival Time Prediction From Brain MRI trains survival prediction from whole MRI volumes without masks. It reaches 0.517 three-class accuracy on BraTS 2019 with saliency maps whose localization Dice is only about 0.20, and it reports that high-Dice models did not necessarily predict survival better. This is a direct warning. If a survival gradient is allowed to shape masks, nothing forces the masks to resemble lesions. That is exactly why we restrict outcome learning to expert-plausible hypotheses.

Automatic Brain Tumour Segmentation and Biophysics-Guided Survival Prediction pulls the other way. It reaches 0.59 accuracy using one hand-designed ratio of nested sub-region volumes (the Relative Invasiveness Coefficient: core volume over whole-tumor volume) plus age. A 162-feature radiomics model reaches only 0.48. We read this as evidence that a few boundary-defined quantities carry prognosis, and that high-dimensional texture features overfit. Our survival head therefore pools a small number of region-level features rather than a full radiomics panel.

Application of the nnU-Net for automatic segmentation of lung lesion on CT images, and implication on radiomic models shows on 899 non-small-cell lung cancer (NSCLC) patients that radiomic survival models built on automatic nnU-Net contours performed comparably to those built on manual ones. Survival was reduced to a binary split at the median, which discards censoring. This paper shows a two-stage pipeline is a credible competitor and supplies our cohort setting.

Two decoupled brain tumor pipelines, Brain Tumor Segmentation and Survival Prediction Using 3D Attention UNet and the biophysics work above, freeze segmentation before fitting survival. The 3D Attention UNet work also treats survival as plain regression in days, ignoring censoring.

For the survival objective we reuse the neural Cox partial likelihood of DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network. As a check against the proportional-hazards assumption, we also use the discrete-time hazard formulation described in Neural Networks to Predict Survival from RNA-seq Data in Oncology.

Research Program

Stage one: learn the space of expert contours. We train a MoSE-style model on LIDC-IDRI, where each lung nodule is traced by four thoracic radiologists. Its $K$ heads should cover the range from tight dense-core contours to inclusive margins. We then freeze the decoders.

Stage two: transfer to survival cohorts. We apply the frozen model to CT scans in the NSCLC survival cohorts. For each patient this yields $K$ hypothesis masks. From these we define the core (the region all hypotheses include) and the band (included by some hypotheses but not all). Lung cancer lesions in these cohorts are larger and more heterogeneous than LIDC nodules, so transfer quality is checked directly. Where the cohorts provide manual contours, we report Dice of the hypotheses against them. If a cohort offers only diameter measurements, we use these as weak anchors for which lesion to analyze, in the spirit of Weakly Supervised Lesion Co-Segmentation on Ct Scans.

Stage three: outcome-trained selection. A shared image encoder produces a feature map. Masked average pooling gives a core feature $\phi_c$, a band feature $\phi_b$, and per-hypothesis features $\phi_k$. The log-risk is

$$r(x) = f_\theta\Big(\phi_c,\ \phi_b,\ \textstyle\sum_{k=1}^{K} \pi_k(x)\,\phi_k,\ a\Big),$$

where $\pi_k(x)$ are the gate weights over the $K$ expert-like hypotheses (summing to one), $a$ is the vector of clinical covariates (age, stage), and $f_\theta$ is a small network.

Training minimizes the Cox partial likelihood. A KL penalty keeps $\pi(x)$ close to the gate MoSE learned on LIDC-IDRI. The decoders never receive the survival gradient, so every mask the survival model uses remains one a radiologist plausibly could have drawn.

Second testbed: BraTS 2019. Here the hypotheses are the nested expert-defined sub-regions (enhancing tumor, tumor core, whole tumor) rather than rater modes. The gate then learns, case by case, which boundary matters. This makes it a learned generalization of the hand-built Relative Invasiveness Coefficient, and it lets us engage the precision-versus-prognosis tension on the benchmark where it was raised.

Analysis. Beyond prediction, we ask whether the survival-preferred hypothesis is systematically the inclusive one, and whether this preference survives across cohorts. That answers, for this setting, the open question of whether infiltrative-margin extent tracks shorter survival.

Baselines

The naive reference point is a linear Cox model on age, stage and consensus lesion volume. Using the same covariates without the volume isolates any imaging contribution. Prior work in the file suggests clinical endpoints dominate outcome variance in NSCLC radiomics (Radiomics-based Prognosis Analysis for Non-Small Cell Lung Cancer attributes 65% of variation to the endpoint), so this floor may be high. Any imaging method must beat it to matter.

The strongest existing competitor on the lung cohorts is the two-stage pipeline of Ferrante et al.: nnU-Net contours, then radiomic and deep features, then a classifier. It reports 0.78 accuracy for above/below-median survival with automatic contours, versus 0.65–0.73 with manual contours. That paper reports no C-index on these cohorts; no reported result was found on Lacuna for C-index on these NSCLC cohorts. We will therefore re-run the pipeline ourselves under censoring-aware evaluation. This means nnU-Net on the same training lesions, a Cox model or random survival forest on the same feature budget, and the same cross-validation folds and hyperparameter search budget as our model. We will also report median-split accuracy so the 0.78 figure remains a reference point.

On BraTS the reference points are 0.59 three-class accuracy from the biophysics model and 0.517 from the mask-free post-hoc model. The BraTS survival validation set has only 29 cases, so we will mainly compare via repeated cross-validation on the survival-labeled training cases, rather than relying on a 29-case number.

The mask-free competitor is an end-to-end 3D CNN trained with the same Cox loss on the same lung crops with no masks, following Hermoza et al. It receives the same encoder and the same tuning budget.

Three ablations isolate the key ingredient. A uniform gate, with $\pi_k$ fixed to the LIDC prior, tests whether outcome-driven selection matters at all. A consensus-only model, using the mean hypothesis mask and no band feature, tests whether the band carries signal. An unconstrained variant lets the survival gradient fine-tune the decoders, testing whether the expert-plausibility restriction matters.

Beating the field means three things. Our model's C-index must exceed the two-stage pipeline and the clinical floor, with bootstrap confidence intervals that exclude no improvement on at least one external cohort. Integrated Brier score (a time-averaged measure of how well predicted survival curves match outcomes) must also improve. And mask plausibility must be preserved, measured as Generalized Energy Distance (how far the predicted mask distribution is from the rater distribution) on held-out LIDC-IDRI and Dice against any available manual contours.

Why not just…?

Why not just average the raters into a consensus mask and run nnU-Net plus a survival model? This is the default practice, and Ferrante et al. show it is not obviously worse than manual contouring. But it discards exactly the quantity we hypothesize is prognostic: how much tissue lies in the disputed margin. The consensus-only ablation and the two-stage baseline settle this directly. If neither the band feature nor the gate improves C-index over them, the hypothesis is wrong for this setting.

Why not just skip masks and train survival end to end from the image? Hermoza et al. show this can predict survival competitively on BraTS. That is a real threat on predictive metrics. However, their saliency localization Dice of about 0.20 means such a model gives no measurable boundary, so it cannot answer whether margin extent tracks risk. It is also not obvious that it would transfer across scanners and sites. We run it with identical encoder and budget. If it matches our C-index externally, we will report that the boundary-structured model's advantage is interpretability and anatomical auditability, not accuracy.

Why not just compute core and band volumes from the transferred hypotheses and feed them to a Cox model, with no learned gate? This is the alternative most likely to match us, and it is cheap. It is our uniform-gate ablation. We think it may well capture most of the gain, since with a few hundred patients a learned gate has limited data to exploit. If it matches the gated model, we will state that the ambiguity band, not outcome calibration, is the useful ingredient, and recommend the simpler pipeline. The gated model is retained only if it improves C-index on an external cohort. A shuffled-outcome control must also show that a gate trained on permuted survival times produces no consistent hypothesis preference.

Why not just let the survival loss fine-tune the segmentation network directly, as DiFF does with diagnosis? With censored outcomes on a few hundred patients, a dense spatial signal is badly underdetermined. The mask-free results suggest the model would drift toward whatever region predicts risk rather than toward the tumor. The unconstrained ablation tests this. We expect it to raise in-cohort C-index slightly while degrading Dice and Generalized Energy Distance and losing more C-index on the external cohort. If instead it generalizes as well with intact masks, the restriction is unnecessary and we will say so.

Decisive Checks

The work stands or falls on four measurements. The first is the C-index gain from adding band features over the consensus-plus-clinical model on an external NSCLC cohort, with bootstrap intervals. The second is the gap between learned and uniform gates on that same external cohort. The third is the trade-off between C-index and mask plausibility for the unconstrained versus restricted models. The fourth is whether the survival-preferred hypothesis is systematically the inclusive one, and whether a permuted-outcome control erases that preference.

On BraTS, the learned gate over nested sub-regions should at least match the hand-built Relative Invasiveness Coefficient under repeated cross-validation. Falling short would indicate that hand-chosen boundary ratios already capture what outcome-driven selection can find. Finally, a discrete-time hazard version of the survival head checks that conclusions do not hinge on the proportional-hazards assumption.