Most imaging-based survival models for cancer are built in two steps. Someone (a radiologist or a segmentation network) first draws a tumor outline in the 3D scan. Features computed inside that outline are then fed to a survival model alongside clinical variables such as age or stage. The weak link is the outline. Experts disagree most at the tumor margin, which is where infiltration, peritumoral vessels, and edema live. There is direct evidence that this disagreement costs prognostic accuracy. When glioma survival models were rebuilt on six different automated segmentations, the finest-grained physiological subregions gave the best mean AUC but were also the least stable across pipelines. Fusing the outlines into a consensus mask did not help, and misclassified short-term survivors had much larger contour errors than correctly classified ones (Overall Survival Prediction of Glioma Patients With Multiregional Radiomics).
We propose to drop the outline entirely. A 3D network would learn where to look from survival outcomes and clinical data alone, anchored only by a single click on the tumor. We would then test a specific, falsifiable hypothesis: the voxels such a model relies on sit disproportionately in the band where observers disagree. If that is true, consensus-contour pipelines are systematically discarding or randomizing the most prognostic tissue. A model that never needs observers to agree should be both more accurate and more stable under changes in who annotates. If it is false, and the attention settles on the agreed-upon core or on a fixed shell around it, we will say so and recommend the cheap fixed-shell recipe instead.
Core Claim
The claim has two parts, and the second is the one that matters.
The first part is methodological. A weakly supervised localizer can be trained end-to-end with a survival loss on anisotropic 3D scans. "Weakly supervised" here means the model learns spatial focus without ever seeing a voxel-level mask. It needs a slice-aware volumetric encoder, a click anchor, and a Cox partial-likelihood objective. Under these conditions it can match contour-based deep radiomics on external survival cohorts.
The second part is empirical. Call the "disagreement band" the voxels on which independent segmentations of the same tumor disagree. The learned attention assigns more mass to this band than its volume share would predict. Hiding the band at test time hurts prognosis more than hiding an equal volume of agreed core.
Together these turn inter-observer variability from a nuisance to be averaged away into a map of where prognostic information is concentrated.
Relation to Prior Work
The closest existing work comes from three directions, and each stops short in a different place.
The first is the diagnosis-first calibration framework (Calibrate the inter-observer segmentation uncertainty via diagnosis-first principle). It learns a weighted fusion of several raters' masks that maximizes a downstream classifier's accuracy, and reports AUC gains of 2.04 to 2.73 percent. It motivates our view that the "right" boundary is defined by downstream utility. It differs from us in three ways: it is 2D, it targets an immediate diagnosis rather than censored time-to-event outcomes, and it requires multiple full rater masks for every training image. Our method needs none, and uses disagreement only as an analysis lens and an evaluation axis.
The second is the radiomics-guided transformer for chest X-rays (Radiomics-Guided Global-Local Transformer for Weakly Supervised Pathology Localization in Chest X-Rays). It closes the loop between attention-derived boxes and radiomic features in 2D. Its finding that tighter attention thresholds hurt classification constrains our design: we use soft attention pooling, not thresholded masks.
The third is the TranSamba encoder (Hybrid Transformer-Mamba Architecture for Weakly Supervised Volumetric Medical Segmentation). It pairs 2D in-plane transformer blocks with a linear-cost state-space model run along the slice axis. It supplies the method we reuse for volumetric context. It is evaluated only against static consensus masks and never against outcomes.
On the prognostic side, the two strongest volumetric survival models make opposite choices, and each leaves a gap. Cross-modality masked pretraining (Cross-Modality Masked Learning for Survival Prediction in ICI Treated NSCLC Patients) fuses whole 3D CT with 22 clinical variables and reaches a C-index of 0.701 for progression-free survival and 0.705 for overall survival. It localizes nothing. Voxel-level deep radiomics (Cross-dataset adaptation of voxel-level deep radiomics for predicting survival in inoperable locally advanced NSCLC treated with immunotherapy) reaches a C-index of 0.73 on an external immunotherapy cohort. It requires manual contours and takes each contour as truth. Our proposal sits between them: it is localized, but contour-free.
Two further lines of work motivate the proposal. Habitat and peritumoral studies such as Habitat-based transformer model in pretreatment 18F-FDG PET imaging for predicting prognosis in cervical cancer: a two-center retrospective study and Novel Radiomic Measurements of Tumor- Associated Vasculature Morphology on Clinical Imaging as a Biomarker of Treatment Response in Multiple Cancers suggest that tissue at and beyond the margin carries prognostic signal. The BraTS synthesis (Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge) shows that near-human segmentation Dice has coexisted with survival accuracy stuck near 62 percent. That pattern is consistent with the outline, not the segmenter, being the bottleneck.
Research Program
The model encodes each scan with a TranSamba-style backbone. Slices are processed at native in-plane resolution by a 2D transformer, and depth is mixed by a linear-time state-space block. This matters because routine CT and MRI slices are several times thicker than in-plane pixels, and full 3D attention on volumes of roughly 512 by 512 by 268 is too expensive.
Each voxel feature $h_v$ receives a nonnegative attention weight $a_v$ that sums to one over the volume. The patient risk score is
$$r = g\Big(\sum_v a_v h_v,\; c\Big),$$
where $c$ is the clinical variable vector and $g$ is a small network that also conditions the attention query on $c$ through feature-wise modulation. This conditioning lets, for example, treatment regimen change where the model looks.
Training minimizes the Cox partial-likelihood loss, which uses censored patients correctly, plus a weak anchor term:
$$\mathcal{L} = \mathcal{L}_{\text{Cox}}(r) + \lambda \sum_v a_v \,\max\big(0,\, d(v,p) - \rho\big).$$
Here $p$ is a single clinician click inside the tumor, $d(v,p)$ is physical distance in millimetres, $\rho$ is a generous radius, and $\lambda$ sets the penalty strength. The anchor exists only to stop attention from drifting to confounders such as body habitus. We will sweep $\rho$, including $\rho=\infty$, and report what happens when the anchor is removed.
We then build disagreement bands. On BraTS 2020, following the multiregional study, we run several independent automated segmenters, including the Isensee 3D U-Net and 3D TransUNet (Beckschen/TransUNet). A voxel belongs to the band if at least one but not all segmenters include it. This is a proxy for human disagreement, and we treat it as such. To validate the proxy, we will collect three independent radiologist contours and three independent clicks on a subset of the external NSCLC test patients. That collection is new data we would gather, not something available in the file.
The core analyses follow from this setup. We compute the ratio of attention mass in the band to the band's volume fraction, per patient and per cohort. We run test-time occlusion: we replace band voxels with local-mean intensity and compare the concordance drop against occluding an equal volume of agreed core. We also test anchor robustness by retraining nothing and simply recomputing predictions from each rater's click.
Baselines
The naive reference point is a Cox model on clinical variables alone, plus the same model with tumor volume from a single contour. We add one more covariate that a skeptic would rightly demand: the per-patient volume of the disagreement band. This last baseline asks whether "how much observers disagree" is itself the prognostic signal. If that single number captures most of our gain, the elaborate attention adds little.
The strongest contour-based competitor is voxel-level deep radiomics on the RTOG 0617 to Tianjin transfer protocol (Cross-dataset adaptation of voxel-level deep radiomics for predicting survival in inoperable locally advanced NSCLC treated with immunotherapy). That paper reports a C-index of 0.73 (95% CI 0.63 to 0.82) on the 150-patient external test set. We will reproduce it on the identical splits: 271 training, 68 internal test, 37 fine-tuning, and 150 external. It gets the same clinical covariates and the same hyperparameter search budget as our model. We will run it once per available contour source, so that its variability across annotators is measured rather than hidden. Beating it means matching or exceeding its external concordance while showing a lower relative standard deviation across annotators.
On BraTS 2020 survival, the reference is the multiregional random-forest pipeline (Overall Survival Prediction of Glioma Patients With Multiregional Radiomics). It reports a mean AUC of 0.73 with relative standard deviation 1.99 for physiological subregions, and 0.71 with 1.39 for anatomical subregions. We will match its survival split and its binarization threshold. Beating it means an AUC at or above 0.73 with stability at or below the anatomical model's 1.39.
The strongest contour-free competitor is a global volumetric fusion model in the spirit of cross-modality masked learning (Cross-Modality Masked Learning for Survival Prediction in ICI Treated NSCLC Patients), which reports 0.701 and 0.705 on its single-center cohort. That cohort is described only in that paper. If it is not accessible, we will reimplement its slice-depth encoder with uniform pooling on our cohorts under the same compute budget, and state that the numbers are not directly comparable.
We also include a 3D, survival-loss extension of diagnosis-first mask fusion, trained on BraTS where multiple segmentations exist. This isolates whether learning from multiple masks beats learning from none.
The key ablations remove one ingredient at a time. We replace the learned attention with uniform attention inside the anchor radius, with attention restricted to the agreed core, and with a fixed peritumoral shell. We replace the cross-slice block with slice-wise pooling. We remove the clinical conditioning of the attention query.
Why not just…?
Why not just use a global 3D encoder with no localization? A global model is immune to contour variability by construction, so it is the most serious threat. Published evidence gives it 0.701 to 0.705 concordance without any localization. If our localized model does not beat it on the external NSCLC test, the accuracy half of our claim fails. The band-enrichment analysis could still stand, though, because it can be run on the global model's own gradient attributions. The comparison therefore settles both questions at once: whether localization pays, and whether any survival-trained model, localized or not, concentrates on the disagreement band.
Why not just draw a fixed peritumoral ring around a single contour? Intratumoral-plus-peritumoral features already work in cervical cancer PET (Habitat-based transformer model in pretreatment 18F-FDG PET imaging for predicting prognosis in cervical cancer: a two-center retrospective study), and a ring is nearly free to compute. This is a plausible winner. We include rings of several widths as ablations. If a ring matches the learned attention in both accuracy and stability, we will report that the disagreement band is prognostic but a fixed shell captures it. That outcome would still be a useful and publishable negative result about learned localization.
Why not just segment with a strong supervised 3D network and ignore the rest? The BraTS synthesis shows segmenters reaching about 0.91 whole-tumor Dice while survival accuracy stayed near 62 percent. The multiregional study found that fusing segmenters with STAPLE did not improve survival prediction. Better outlines alone therefore have a documented record of not moving prognosis. The voxel-level radiomics baseline, run once per contour source, directly measures how much annotator choice moves its predictions.
Why not just fuse the multiple masks with a survival-weighted scheme, as the diagnosis-first framework does? That approach requires several full 3D rater masks per training patient. Such masks exist for almost no survival cohort, which is why we only run it on BraTS as a baseline. If it clearly beats our click-anchored model there, we will conclude that multi-rater masks are worth collecting for prognosis. That conclusion bears on the value of the method, not on the band hypothesis.
Proposed Experiments and Decisive Checks
All experiments use BraTS 2020 survival cases and the RTOG 0617 to Tianjin non-small cell lung cancer transfer protocol. The ICI cohort from the masked-learning paper is added only if it can be obtained. Metrics are Harrell's concordance index, time-dependent AUC at one and two years, Kaplan-Meier separation with log-rank tests, and relative standard deviation of predictions across annotation sources. For our model the annotation sources are clicks; for baselines they are contours.
Three results would be decisive. First, band enrichment: an attention-mass-to-volume ratio significantly above one in both diseases, holding for automated-segmenter bands and for the radiologist-contour subset. Second, asymmetric occlusion: hiding the band lowers external concordance more than hiding equal-volume core. This effect must hold for the contour-based baseline too, not only for our model. Third, stability: on the radiologist subset, our model's prediction spread across three clicks is smaller than the voxel-radiomics baseline's spread across three contours, at matched or better concordance.
If enrichment fails, the core claim is refuted even if accuracy is good. If enrichment holds but a fixed ring matches our accuracy, the finding stands and the learned localizer is unnecessary. Only if all three checks pass and the ring and global baselines are beaten does the full proposal hold.
References
- Overall Survival Prediction of Glioma Patients With Multiregional Radiomics
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Radiomics-Guided Global-Local transformer for weakly supervised pathology localization in chest X-Rays
- Hybrid Transformer-Mamba Architecture for Weakly Supervised Volumetric Medical Segmentation
- Cross-Modality Masked Learning for Survival Prediction in ICI Treated NSCLC Patients
- Cross-dataset adaptation of voxel-level deep radiomics for predicting survival in inoperable locally advanced NSCLC treated with immunotherapy
- Habitat-based transformer model in pretreatment 18F-FDG PET imaging for predicting prognosis in cervical cancer: a two-center retrospective study
- Novel Radiomic Measurements of Tumor- Associated Vasculature Morphology on Clinical Imaging as a Biomarker of Treatment Response in Multiple Cancers
- Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge