Cheap clinical variables already predict cancer survival well. Stage, age, nodal status and surgical margins give nomograms with discrimination above 0.8. A deep model reading a digitized pathology slide costs orders of magnitude more per patient, because a gigapixel image has to be tiled and pushed through a vision network. It also adds prognostic value only for some patients. We propose a two-tier stacked survival model. The first tier is a clinical model trained on the population-scale SEER cancer registry. The second tier is a slide model trained with no pixel annotations at all. Between them sits a small learned gate that decides, from the clinical record alone, whether reading the slide is worth its cost for this patient.
The key design choice is what the gate predicts. Standard early-exit cascades stop when the cheap model is confident. In survival analysis that signal is misleading. A patient whose clinical model says "50% chance of surviving five years" may be correctly and irreducibly 50%, and no slide will change that. We instead train the gate to predict improvability: the expected drop in a censoring-aware survival loss if the slide model's output were added. The claim is that improvability, not uncertainty, is the right signal for spending expensive compute in time-to-event prediction. We also claim it can be learned from a few hundred multimodal patients once the clinical tier has been fixed on hundreds of thousands of registry records.
Core Claim
Consider a clinical survival model trained on SEER and a slide model trained only on slide-level survival outcomes. A gate that regresses per-patient loss improvement will recover most of the full two-modality accuracy gain while running the slide model on a minority of patients. At every matched deferral rate, it will do better than deferring on clinical uncertainty, on intermediate stage, or on cascade confidence.
We define the target explicitly. Let $x_i$ be patient $i$'s clinical record, $z_i$ their slide, and $y_i$ their possibly censored survival outcome. Let $\ell$ be the inverse-probability-of-censoring-weighted Brier score at a fixed horizon, meaning squared error on "alive at five years" with weights that correct for patients lost to follow-up. Let $f_{\text{clin}}$ be the clinical predictor and $f_{\text{stack}}$ the stacked predictor that also uses the slide. The per-patient gain is
$$\Delta_i = \ell\big(y_i, f_{\text{clin}}(x_i)\big) - \ell\big(y_i, f_{\text{stack}}(x_i, z_i)\big).$$
The gate $g(x_i)$ is fit out-of-fold to predict $\Delta_i$ from the clinical record and the clinical tier's output alone, since the decision must be made before the slide is processed. We read the slide when
$$g(x_i) > \lambda c,$$
where $c$ is the slide model's per-patient cost and $\lambda$ sets the exchange rate between cost and accuracy. Individual $\Delta_i$ values are very noisy. But $g$ only has to estimate their conditional mean, and regression averages the noise out. A low-capacity gate such as a monotone gradient-boosted model or a small spline regression should therefore suffice.
Relation to Prior Work
The mechanism extends Unifying Stacking and Cascading for Efficient Ensemble Inference (LazyStack), the closest existing work. LazyStack orders base models with a Markov decision process and exits once small "substacker" networks become confident on the accumulated predictions. It reports a 38x speedup on NSL-KDD at 97% of full-ensemble accuracy. It is formulated for classification only, and it documents that confidence-based exits break down when probability mass spreads across many classes. Survival prediction has an analogous problem: a well-calibrated but intermediate risk looks unconfident even when nothing more can be learned. We keep LazyStack's idea of aggregating the evidence accumulated so far. We replace its confidence exit with a regression on improvability.
We also argue that its ordering search mostly becomes unnecessary here. With two tiers whose costs differ by several orders of magnitude, the only real decision is whether to defer. That simplification is testable, and we test it.
Several papers motivate where the gain should live. Ensemble Learning of Colorectal Cancer Survival Rates found that combining TNM staging with learned models raised accuracy for ambiguous stage II/III patients from 59–65% to 79.2–82.5%, but only on patients where the models agreed. This suggests value is concentrated in a subgroup that a rule might or might not capture. Ten-year Survival Prediction for Breast Cancer Patients found the opposite pressure. Seventeen clinical variables beat 24,367 genomic features on METABRIC (XGBoost AUC 70.84% versus 67.12%). Expensive modalities can therefore be net harmful when applied to everyone, which is exactly the case for selective use. On the registry side, A coupling approach of a predictor and a descriptor for breast cancer prognosis and SEER-based survival nomogram for salivary acinar cell carcinoma using clinical and surgical factors show how strong a tabular-only tier can be on SEER. The first reports co-training AUC 0.81 on 162,500 records. The second reports a nomogram C-index of 0.824.
Other work constrains the design. Gaussian Meta-Space Augmentation for Stacking Ensembles in Multimodal IPMN Risk Stratification shows that multimodal stacking beats end-to-end 3D networks on a 678-patient cohort. It also shows that high-capacity combiners overfit small cohorts and need meta-space augmentation, and that the augmentation does little for linear combiners. Our multimodal cohort is similarly small, so we use a low-capacity combiner and gate. Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap warns that ensembling can hurt calibration. We therefore report calibration separately and choose a loss that penalizes miscalibration.
On annotation, most label-efficient pathology work still requires some manual signal. HistoSegNet uses patch-level tissue tags, Learning to Segment Medical Images with Scribble-Supervision Alone uses scribbles, and Robust Histopathology Image Analysis: To Label or to Synthesize? builds a synthetic pipeline because, as it notes, 50 annotated patches can take a pathologist over 200 hours. We follow the patient-level-label route of Domain Adaptation for Ulcerative Colitis Severity Estimation Using Patient-Level Diagnoses. That paper supervises frame-level predictions with only a patient-level diagnosis. We supervise a multiple-instance slide model, which pools tile embeddings into one slide prediction, with only the patient's survival outcome.
Research Program
Public SEER data contains no images. We treat this as a structural fact of the design rather than hide it. The clinical tier is a discrete-time survival model (gradient-boosted hazards, alongside a Cox baseline) trained on SEER. It uses the variables SEER shares with the TCGA clinical tables: age, sex, stage, grade, histology code, nodal status and surgery code. We fit it separately for two or three cancer types for which TCGA offers enough slides with follow-up. TCGA is not a SEER catchment sample, so we measure the resulting shift directly. We compare the SEER-trained tier against a clinical tier trained only on TCGA, and we recalibrate the SEER tier on TCGA training folds.
The slide tier is an attention-pooling multiple-instance survival model over tile embeddings from a frozen pathology encoder. It is trained on TCGA slides with only the patient's time and event as supervision, so no pixel, region or patch labels are used.
A linear stacker combines the clinical risk, the slide risk and their interaction on out-of-fold predictions. The gate is then fit on out-of-fold $\Delta_i$ values from nested cross-validation within TCGA, so no patient's gate target comes from a model that saw that patient.
We also analyse which patients the gate selects: what fraction are stage II/III, whether selection tracks the clinical model's variance, and whether the selected set is stable across folds. Two purposes follow. First, the analysis tells us whether the gate has found something a clinician's rule would not. Second, the gate's ranking doubles as a triage list for scarce pathologist review, which ties improvability to the annotation bottleneck. Where expert attention is limited, it should go to patients whose slides change the prognosis.
As a second, image-free testbed we use METABRIC, with the same construction. The clinical variables form the cheap tier and genomic profiles form the expensive tier. This checks that the finding does not depend on the pathology encoder. It is also a hard case, because on METABRIC the expensive modality loses on average.
Baselines
The naive reference is the clinical tier alone, which costs nothing extra. This SEER-trained survival model is evaluated on the same TCGA test folds. Its SEER-scale analogue in the file is the co-training AUC of 0.81 from A coupling approach of a predictor and a descriptor for breast cancer prognosis. That figure is on a different task definition, so it only anchors the tabular regime and is not a number to beat. On METABRIC the reference is clinical XGBoost at 70.84% AUC, as reported by Ten-year Survival Prediction for Breast Cancer Patients. No reported result was found on Lacuna for multimodal clinical-plus-slide survival on TCGA, so the upper reference is our own always-run stack. We also train a single end-to-end multimodal network on identical folds.
The strongest existing competitor is LazyStack from Unifying Stacking and Cascading for Efficient Ensemble Inference. We adapt it by recasting five-year survival as censoring-weighted binary classification, then training its substacker and confidence exit on the same two tiers. It is the strongest competitor because it is the only method in the file that already combines stacking with cost-aware exits. It has no reported survival result.
Two simpler deferral rules complete the field. The first defers on clinical uncertainty, measured as the width of a bootstrap interval on predicted five-year survival. The second defers all stage II/III patients, which is the rule the Nottingham result points to.
The key-ingredient ablation holds everything fixed: the same features, the same gate model class, and the same out-of-fold protocol. Only the gate's target changes, from predicted $\Delta_i$ to predicted uncertainty. A second ablation swaps the SEER-trained clinical tier for a TCGA-only one to isolate what population scale contributes.
The fairness conditions are as follows. All methods share the frozen tile embeddings, patient-level folds and hyperparameter search budget (the same number of configurations per method). Every method is scored at matched deferral rates of 10%, 20%, 30% and 50%. "Beating" a competitor means achieving a lower integrated Brier score and a higher C-index at the same deferral rate, with fold-bootstrap intervals that exclude zero difference. It also means recovering at least half of the always-run stack's gain over clinical-only at 30% deferral.
Why not just…?
Why not just use the clinical model and never read slides? This alternative wins if the always-run stack does not beat clinical-only. The METABRIC result shows that outcome is plausible for an expensive modality. Our first experiment tests exactly this. If the full stack's gain is not significant on a cancer type, we report that the gate has nothing to find there and drop that cancer type. The claim concerns where gain exists, not that it always exists.
Why not just defer on uncertainty, or on stage II/III? This is the real threat. Uncertainty and improvability may be strongly correlated, and the stage rule captures the subgroup the Nottingham study highlighted. The target-swap ablation settles the question directly. If uncertainty-based deferral matches the learned gate at every deferral rate, the core claim is false and we will say so. We will then report the simpler rule as the recommended practice. The gate-composition analysis will show whether selection reduces to stage.
Why not just train one multimodal network or distill the stack? Distillation compresses the head, but in this setting the cost comes from tiling and encoding the slide, and any student that uses the slide must still pay it. Collaborative Learning for Deep Neural Networks and Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation cut model cost, not input-acquisition cost. The IPMN study found end-to-end networks overfit small multimodal cohorts. We include the end-to-end model on identical folds anyway, as the always-run comparison.
Why not just run LazyStack unchanged? It needs a confidence signal, and censored survival outcomes supply only a calibrated risk, whose extremes do not coincide with where images help. The adapted LazyStack baseline measures the practical cost of that mismatch.
Proposed Experiments
The first check is gating-free. On each TCGA cancer type and on METABRIC, does the always-run stack beat clinical-only in integrated Brier score and C-index, and how is the oracle $\Delta_i$ distributed? If a small fraction of patients carries most of the gain, selective reading has room to work. If the gain is spread evenly, no gate can do much better than random deferral.
The decisive experiment traces accuracy-versus-deferral curves for the learned gate, uncertainty deferral, stage deferral, adapted LazyStack and random deferral. The main metric is area under the curve of IPCW Brier improvement against the fraction of slides processed. Secondary metrics are time-dependent five-year AUC, calibration slope, and GPU-seconds per patient.
The central hypothesis is that the learned gate recovers at least half of the full gain while processing 30% or fewer of slides, and that it beats uncertainty deferral at matched rates. If uncertainty deferral ties it, that falsifies the claim.
Two further checks close the program. The SEER-versus-TCGA-only comparison of the clinical tier tests whether population scale makes the gate's job easier by leaving a cleaner residual. The ordering check asks whether an MDP search over tier order ever selects anything other than clinical-first under this cost asymmetry. That answers the open ordering question from LazyStack for the case of extreme cost differences between tiers.
References
- Unifying Stacking and Cascading for Efficient Ensemble Inference
- Ensemble learning of colorectal cancer survival rates
- Ten-year Survival Prediction for Breast Cancer Patients
- A coupling approach of a predictor and a descriptor for breast cancer prognosis
- SEER-based survival nomogram for salivary acinar cell carcinoma using clinical and surgical factors
- Gaussian Meta-Space Augmentation for Stacking Ensembles in Multimodal IPMN Risk Stratification
- Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap
- HistoSegNet: Semantic Segmentation of Histological Tissue Type in Whole Slide Images
- Learning to Segment Medical Images with Scribble-Supervision Alone
- Robust Histopathology Image Analysis: To Label or to Synthesize?
- Domain Adaptation for Ulcerative Colitis Severity Estimation Using Patient-Level Diagnoses
- Collaborative Learning for Deep Neural Networks
- Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation