Cancer staging sorts patients mostly by how big the tumor is and where it has spread. Two patients with the same size and stage can still have very different outcomes, because size says little about how aggressively the tumor invades surrounding tissue. Radiologists already record a cheap, routine trace of this invasion without meaning to. When two readers measure the same tumor with the standard two-line "crosshair" (the longest diameter and its perpendicular, as required by the RECIST response criteria), they agree on well-circumscribed lesions and disagree on lesions with blurry, infiltrative edges. Current machine learning treats that disagreement as noise to be averaged away. We propose to treat it as signal.
The plan has three steps. First, collect several readers' crosshairs per tumor, which takes seconds per case. Second, train a segmentation network that predicts, for every pixel, how likely a random expert is to count it as tumor. Third, test whether the width and content of the predicted "ambiguity band" predicts survival beyond anatomical stage and tumor volume. The trained model then estimates margin ambiguity on new scans with no reads at all. If this works, cheap sparse annotations would supply something that currently requires multiple dense expert contours: a reproducible, image-derived measure of boundary indistinctness that staging systems ignore.
Core Claim
The spatial pattern of inter-reader disagreement on sparse caliper annotations can be learned without dense masks. Once learned, it carries prognostic information that is not explained by stage, tumor volume, or a single network's own uncertainty. The claim connects the three directions this way. Inter-observer variability is the supervisory signal. The annotation bottleneck sets the constraint, so only crosshairs are used. Anatomical staging is the target the resulting feature must beat.
Where This Sits
The closest prior work falls into three groups, and none does this.
The first group learns from RECIST crosshairs to avoid dense annotation, and it supplies our method. Weakly Supervised Lesion Co-Segmentation on Ct Scans turns DeepLesion crosshairs into GrabCut pseudo-masks and reaches 90.3% Dice. RECIST Weakly Supervised Lesion Segmentation via Label-Space Co-Training turns each crosshair into an inner quadrilateral and an outer circle and trains two networks to agree in the ring between them, reaching 0.907 Dice on KiTS19. Both assume a single, correct crosshair. In the second paper the crosshairs were simulated from ground-truth masks, so no human disagreement exists in its data. We reuse the quadrilateral and circle construction but apply it once per reader.
The second group models multi-rater disagreement, and it motivates the proposal. The Latent Doctor Model for Modeling Inter-Observer Variability and Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation learn distributions over expert labels or per-rater latent styles. More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel shows that keeping individual raters instead of a consensus cuts calibration error from 0.183 to 0.050. Deep learning-based GTV contouring modeling inter- and intra- observer variability in sarcomas predicts six-level reader-agreement maps for sarcoma tumor volumes. These papers establish that disagreement is structured and learnable. Every one of them needs dense multi-rater labels, and the sarcoma study needed four readers contouring each case three times. None asks whether the disagreement predicts outcome. Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle does tie raters to a clinical endpoint, but in the opposite direction from us. It weights dense rater masks by how useful they are for a binary diagnosis and so resolves the disagreement. We keep the disagreement and use it as a time-to-event predictor.
The third group replaces anatomical staging with image-derived scores, and it supplies the bar we must clear. Automated and Interpretable Survival Analysis from Multimodal Data reaches a C-index (the probability that the model ranks a random pair of patients' survival times correctly) of 0.838 on head and neck CT plus clinical data, against 0.614 for TNM staging. An MRI radiomics approach using invasion-based weak supervision for identifying and evaluating aggressive PitNETs uses the extreme categories of an anatomical invasion scale as weak labels, discards the equivocal middle where readers disagree most, and still depends on dense nnU-Net segmentations. Non-invasive decision support for NSCLC treatment using PET/CT radiomics finds prognostic signal in the 10–20 mm peritumoral rim, which supports our hypothesis that the margin is where biology hides. RECIST 1.1 assessments variability: a systematic pictorial review of blinded double reads documents that real readers choose different axes and endpoints by millimeters. It offers no model. We take its catalogue as evidence that the variation we want to learn exists in routine practice.
Research Program
Data collection is deliberately light. On the RADCURE head and neck cohort, three or more radiologists independently place RECIST-style crosshairs on the axial slice of maximal extent of the primary tumor for roughly 600 training and 300 test patients, drawn within the cohort's published training and external-test split. At a few seconds per read this costs a few expert-hours, not weeks. On the MGH and TCIA sarcoma data from the GTV contouring study, we simulate each reader's crosshair from each of the twelve existing contours (four readers times three trials). This gives real human margin variability with dense agreement maps held out for validation. KiTS19 and DeepLesion serve only to check that single-reader segmentation quality matches published label-space co-training.
The model extends label-space co-training to many readers. Each reader $r$ contributes an inner quadrilateral $Q_r$ and an outer circle $C_r$. The intersection of all $Q_r$ is treated as certain tumor and the region outside every $C_r$ as certain background. The network predicts a per-pixel inclusion probability $p(x)$, read as the chance that a random expert would count pixel $x$ as tumor. In the zone where readers' quadrilaterals disagree, a calibration loss pushes $p(x)$ toward the empirical fraction of readers whose quadrilateral covers $x$. Elsewhere, the original region-constrained consistency loss between two co-trained networks applies. A small per-reader offset, in the spirit of Pionono's rater latents, absorbs systematic over- or under-measurement by individual readers. Without it, "reader A always measures generously" would be mistaken for tumor ambiguity. The ambiguity band is the set of pixels with $0.2 < p(x) < 0.8$. Its descriptors are band width normalized by tumor size, band asymmetry around the circumference, and intensity heterogeneity inside the band.
At test time the model runs on scans with no reads. Its descriptors enter a Cox survival model, which relates covariates to the instantaneous risk of death, alongside TNM stage, HPV status, age, and the core tumor area.
Two confounds are built into the analysis. Margin ambiguity in head and neck CT can come from dental metal artifacts or poor contrast timing rather than from invasion, so an artifact score and contrast phase are entered as covariates. HPV-positive tumors are known to be both better circumscribed and more favorable, so all incremental analyses are stratified by HPV status.
Baselines
The naive reference is TNM staging alone, for which Automated and Interpretable Survival Analysis from Multimodal Data reports a C-index of 0.614 on RADCURE. We add a stronger naive reference of TNM plus clinical variables plus crosshair-derived size, meaning the product of mean reader diameters. This is the information a clinic already has. Our descriptors must improve on it in a likelihood-ratio test and by C-index on the held-out test patients.
The strongest existing method is the MultiFIX approach from the same paper: a 3D DenseNet on CT combined with health-record variables, reported at C-index 0.838. We retrain it on our exact train and test patients with the same hyperparameter search budget, because the test subset will differ from the original 670. Its reported figure is a reference, not the comparison. We do not expect a handful of ambiguity descriptors to beat a full deep image model outright. The success criterion is that adding our descriptors to the retrained MultiFIX raises its C-index by a statistically supported margin (bootstrap confidence interval excluding zero). A second success criterion is that the descriptors alone recover a substantial share of the gap between 0.614 and MultiFIX while being a single, auditable measurement.
For segmentation, the baselines are single-reader label-space co-training, reported at 0.907 Dice on KiTS19 by RECIST Weakly Supervised Lesion Segmentation via Label-Space Co-Training, and GrabCut, reported at 0.891 there. We reproduce both on KiTS19 to confirm our implementation. On the sarcoma data the reference is the densely supervised agreement-map model of Deep learning-based GTV contouring modeling inter- and intra- observer variability in sarcomas, with continuous Dice 86.8% against reader confidence maps and human-to-human agreement of 90.5%. Our model, trained on crosshairs only, beats expectations if it comes within a few points of 86.8% on that metric, restricted to the 2D maximal slice and evaluated on the same patients.
The ablation that isolates the key ingredient has three arms, all with identical architecture, data, and compute. The first trains on one reader's crosshair per case and takes Monte Carlo dropout variance as the uncertainty. The second trains on all readers' crosshairs pooled as independent samples, with no calibration loss. The third is the full model. If the first arm's uncertainty band predicts survival as well as the third's, then reader disagreement is not the ingredient that matters.
Why not just…?
Why not just record how much the two readers' diameters differ and use that number as a covariate? This is the most serious alternative. On cases that happen to have multiple reads, it may capture most of the prognostic signal. It fails in two ways our experiments measure. First, it is unavailable for any patient read once, which is nearly every patient outside a trial. Second, a scalar diameter difference mixes axis-choice disagreement with endpoint disagreement, which van der Vegt et al. describe as distinct failure modes. We report the raw-disagreement covariate's C-index on the multi-read test patients. If it matches the model's prediction on those same patients, the contribution narrows to a deployment one: estimating that quantity on single-read scans. We would state that honestly.
Why not just use the segmentation network's own uncertainty? Model uncertainty reflects what the network has not seen. Reader disagreement reflects what experts find visually indistinct. The two need not coincide. Can et al. in Learning to Segment Medical Images with Scribble-Supervision Alone use Monte Carlo dropout only to filter pseudo-labels, not as a phenotype. The first ablation arm tests this directly, both against the held-out sarcoma agreement maps and on survival.
Why not just train an end-to-end deep survival model and let it find the margin itself? MultiFIX's 0.838 shows such models are strong, and they may already encode margin texture implicitly. If our descriptors add nothing on top of a retrained MultiFIX, the honest conclusion is that their value is interpretability and label efficiency, not new prognostic information. We test the label-efficiency side by training both pipelines on 10%, 30%, and 100% of outcome-labelled patients. This matters because the ambiguity model learns from crosshairs, which do not require follow-up.
Why not just collect a majority-vote or STAPLE consensus and segment that? Consensus is the step that destroys the signal we want. Grove et al.'s calibration result and the Latent Doctor Model's finding that majority-vote training loses ambiguity information both show this. A consensus-crosshair arm is included and should match the single-reader arm.
Proposed Experiments
Four checks decide the project.
On the sarcoma data, the first check is whether the crosshair-trained inclusion map matches the dense twelve-contour agreement maps. The metrics are continuous Dice and the Spearman correlation between predicted and observed per-pixel agreement. This establishes that sparse annotations can recover what Marin et al. needed three rounds of four-reader contouring to obtain.
On held-out RADCURE patients with multiple reads, the second check compares predicted band width with observed reader disagreement. The Latent Doctor Model's entropy correlation of 0.728 on ISUP grading is a loose reference from a different task, not a target.
The third check is incremental prognostic value. We measure the C-index and likelihood-ratio gain over TNM plus clinical variables plus size, then over retrained MultiFIX, stratified by HPV status and adjusted for artifact score.
The fourth check is closest to the motivating problem. Within patients sharing the same T stage and tumor size tertile, we ask whether the ambiguity band separates Kaplan–Meier survival curves (log-rank test). This addresses the case staging cannot resolve: anatomically identical tumors with divergent outcomes.
The hypothesis fails if the dropout-uncertainty arm performs as well as the reader-disagreement arm, or if any survival gain vanishes after the artifact covariate is added. The main scope limit is that everything here is 2D on the maximal slice. Extending ambiguity bands through the volume is deferred until the 2D signal is shown to exist.
References
- Weakly Supervised Lesion Co-Segmentation on Ct Scans
- RECIST Weakly Supervised Lesion Segmentation via Label-Space Co-Training
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel
- Deep learning-based GTV contouring modeling inter- and intra- observer variability in sarcomas
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Automated and Interpretable Survival Analysis from Multimodal Data
- An MRI radiomics approach using invasion-based weak supervision for identifying and evaluating aggressive PitNETs
- Non-invasive decision support for NSCLC treatment using PET/CT radiomics
- RECIST 1.1 assessments variability: a systematic pictorial review of blinded double reads
- Learning to Segment Medical Images with Scribble-Supervision Alone