After a tumor is surgically removed, doctors must decide whether to give extra treatment, such as chemotherapy, radiotherapy or immunotherapy, to kill cancer cells that may remain. A wave of recent models combines pathology slides, CT scans and clinical records to predict who will relapse. A patient flagged as high-risk is then assumed to need more treatment. Predicting relapse, however, answers a different question from predicting benefit. A high-risk tumor may be resistant to the drug, and an intermediate-risk tumor may respond strongly. This proposal builds and tests a multimodal model whose target is the patient's expected gain from adjuvant therapy, not their risk. It uses a design meant to survive the small event counts typical of surgical cohorts.
The question is sharp, and the answer is not foregone. Every multimodal model in the surveyed literature estimates prognosis under usual care. When these models show that "high-risk patients benefit," that pattern may only restate the arithmetic of constant relative effects: a fixed hazard ratio produces larger absolute gains in patients with higher baseline risk. The proposal asks whether images and slides carry information about relative treatment response that clinical stage and a risk score do not. It answers with an estimator that is anchored to clinical stage, kept small, and protected against biased treatment selection, so that the answer can come out either way.
Core Claim
Multimodal oncology models should be judged by how well they rank patients by treatment benefit. The proposal makes two claims. First, a benefit model can be built in which clinical variables carry the main effect and frozen pathology and CT embeddings contribute only a small, shrunk correction, and this model can be trained on cohorts with tens to a few hundred events. Second, the decisive empirical test is whether image-derived features improve benefit ranking beyond a "risk magnification" model, in which benefit is a fixed proportional reduction of each patient's predicted risk. If they do not, the field's current practice of choosing patients by risk cutpoints is justified. If they do, risk cutpoints are misallocating therapy, and we can say which patients are affected.
Relation to Prior Work
The closest work is the gastric cancer model of Interpretable Multimodal Fusion Model Enhances Postoperative Recurrence Prediction in Gastric Cancer. It combines a CT radiomics score, a pathology score and clinical stage by logistic regression into a recurrence risk score. It then shows post hoc that patients above a cutoff of 0.19 benefit from adjuvant chemotherapy and those below do not. The breast cancer model of Multimodal deep learning with routine clinical data for recurrence risk stratification in HR+/HER2- early breast cancer uses a hierarchical cross-attention transformer over slides, ultrasound, report text and clinical data. It separates relapse risk sharply, with a hazard ratio of 6.06 between its high- and low-risk groups. Both motivate this proposal, and both stop at prognosis. Neither is trained on, or validated against, benefit. Neither separates "benefits because high-risk" from "benefits because responsive."
Classical oncology has the same structure. The nomograms in esophageal cancer (Survival risk stratification based on prognosis nomogram to identify patients with esophageal squamous cell carcinoma who may benefit from postoperative adjuvant therapy, Nomogram-based risk stratification to analyze the value of receiving postoperative adjuvant therapy after neoadjuvant immunochemotherapy for patients with locally advanced esophageal squamous carcinoma) and the circulating tumor DNA algorithm for small-cell lung cancer (Integrated circulating tumor DNA-based prognostic algorithm for limited stage small-cell lung cancer under definitive chemoradiotherapy and utility in consolidation immunotherapy benefit prediction) stratify by risk and then compare treated against untreated patients within each tier. They provide the clinical template we inherit. They also supply the benchmark pattern we must beat: benefit appears in the high tier only.
The architecture line constrains our design more than it supplies it. ME-Mamba (ME-Mamba: Multi-Expert Mamba with efficient knowledge capture and fusion for multimodal survival analysis) fuses slides and genomics with linear-cost state space models and optimal-transport alignment. CA-MLIF (CA-MLIF: Cross-Attention and Multimodal Low-Rank Interaction Fusion Framework for Tumor Prognostic Prediction combines cross-attention with low-rank tensor fusion. SlotSPE (Structural Prognostic Event Modeling for Multimodal Cancer Survival Analysis) compresses inputs into "prognostic event" slots. PIBD (Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction) discards 60 to 75 percent of slide patches through a prototype bottleneck. All four report survival concordance on TCGA prognosis tasks, and none touches treatment.
The negative result that shapes everything here is Robustness of Multimodal Foundation-Model Forecasting for Postoperative Cancer Outcomes. On 466 resected lung cancers with 56 recurrences, the best pathology-plus-CT foundation-model fusion reached a C-index (the probability that a model orders a random pair of patients correctly by event time) of 0.799. Pathologic stage alone reached 0.837. On a 198-case kidney cohort the fusion model collapsed to 0.617 against a clinical 0.815. In the same study, unweighted late averaging beat learned stacking. The lesson for this proposal is direct. Benefit estimation is statistically harder than prognosis, because it depends on the difference between two noisy arms. Any deep fusion head trained to estimate benefit on cohorts of this size will overfit unless clinical stage is the anchor and the image contribution is small and shrunk. We therefore borrow the anchored-residual idea from that paper and adopt late, low-capacity fusion. We deliberately avoid the cross-attention and tensor heads of MRRP and CA-MLIF.
Research Program
The estimator works with restricted mean survival time up to a horizon $t^*$ (for example, five years), which is the expected disease-free time within that window. Benefit is $\tau(x)=\mathbb{E}[T_1\wedge t^* - T_0\wedge t^* \mid X=x]$, where $T_1$ and $T_0$ are the patient's disease-free times with and without adjuvant therapy. This target is in months, so clinicians can weigh it against toxicity, and it does not assume proportional hazards. Censoring is handled by inverse-probability-of-censoring-weighted pseudo-outcomes $\tilde Y_i$, one per patient. These are computed from a censoring model fitted on clinical variables.
Benefit is learned with a residual-on-residual loss, the R-learner form, which removes the prognostic signal before fitting effects:
$$\hat\tau=\arg\min_{\tau}\sum_i\Big(\tilde Y_i-\hat m(x_i)-\big(A_i-\hat e(x_i)\big)\,\tau(x_i)\Big)^2+\lambda\,\Omega(\tau).$$
Here $A_i$ indicates whether patient $i$ received adjuvant therapy. $\hat m(x)$ is a cross-fitted prediction of the outcome ignoring treatment, and $\hat e(x)$ is a cross-fitted propensity model, meaning the probability of receiving therapy given the patient's features. $\Omega$ is a penalty described below. The prognostic model $\hat m$ may use the full multimodal input, because it only needs to predict outcomes. The benefit function is deliberately constrained:
$$\tau(x)=\tau_{\text{clin}}(z)+\textstyle\sum_k \gamma_k\, s_k(h_k).$$
In this decomposition, $z$ is the clinical vector (stage, grade, nodal ratio, margin). $\tau_{\text{clin}}$ is a small additive model. $h_k$ is the frozen embedding of modality $k$: a slide encoder such as UNI or TITAN, or a CT encoder such as CT-FM or Pillar-0, as used in the cited studies. $s_k$ is a one-dimensional linear projection of $h_k$. The coefficients $\gamma_k$ are ridge-shrunk toward zero, and $\Omega$ penalizes them more heavily than the clinical terms. This is late, score-level fusion applied to benefit rather than to risk. It mirrors the variance control that made late averaging win in the Wu et al. lung study. When the data are weak, the model falls back to a clinical benefit model rather than to noise.
Within slides, we test one optional addition: a PIBD-style patch filter that keeps the top 25 to 40 percent of tiles before pooling. We adopt it only if it improves held-out benefit ranking.
Confounding by indication is the main threat to validity. Sicker-looking patients get more treatment, and pathology slides may encode exactly what drove the prescribing decision. We handle this in three ways. The propensity model includes the image projections, so treatment channeling visible in the slides is adjusted for. We restrict analysis to the region where the estimated propensity lies between 0.1 and 0.9, so that we compare only patients who could plausibly have gone either way. And we report how strong an unmeasured confounder would have to be to explain away each estimated effect modification. Before any modeling, we check whether treated and untreated patients overlap enough. If adjuvant chemotherapy is near-universal in a cohort, the overlapping subset may be too small and that cohort is reported as unusable for benefit estimation.
Validation has two tiers. The first is semi-synthetic. We take TCGA slide and genomic cohorts already used by ME-Mamba, SlotSPE and PIBD and simulate treatment assignment that depends on clinical and slide features. We also simulate benefit under three regimes: benefit proportional to baseline risk only, benefit driven by a slide-derived feature independent of risk, and no heterogeneity. Because true benefit is known, we can check that the estimator detects image-driven effect modification when it exists. We can also check that it does not hallucinate such modification when benefit simply tracks risk. The second tier is real clinical data. The primary cohorts are those where adjuvant therapy was given selectively and the modalities exist: the multicenter locally advanced gastric cancer cohort with its external validation sites and the NCT02555358 trial cohort from the gastric cancer study, and resected esophageal squamous cohorts like those behind the nomogram papers. A resected lung cohort comparable to Wu et al.'s supplies the low-event stress test. These are held by the cited groups, so data access through collaboration is a real dependency and is stated as one.
Baselines
The naive reference point is to treat everyone or no one. This gives zero benefit-ranking ability by construction. It anchors the policy-value comparisons, measured as expected disease-free months gained per patient treated under each allocation rule.
The clinical-anchor baseline is a T-learner on clinical variables alone: separate restricted-mean-survival models for treated and untreated patients, using stage, grade and nodal status, fitted with the same cross-fitting and overlap restrictions as our model. This baseline represents the nomogram tradition. Given Wu et al.'s finding that stage alone reached a C-index of 0.837 against 0.799 for the best fusion in Robustness of Multimodal Foundation-Model Forecasting for Postoperative Cancer Outcomes, it is likely the hardest baseline on small cohorts.
The strongest existing multimodal method is RSA-style risk stratification from Interpretable Multimodal Fusion Model Enhances Postoperative Recurrence Prediction in Gastric Cancer. Its reported recurrence discrimination is AUC 0.903 in training, 0.902 in internal validation and 0.884 to 0.889 in external validation. We will refit it on the same splits with the same frozen encoders and use its score in two ways. The first is the published cutpoint rule. The second, and stronger, is the "risk magnification" model: benefit equals a single fitted proportional effect applied to RSA-predicted risk. Every patient-specific benefit estimate then comes entirely from prognosis.
For prognosis, the deep-fusion comparator is an MRRP-style hierarchical cross-attention model (Multimodal deep learning with routine clinical data for recurrence risk stratification in HR+/HER2- early breast cancer, C-index 0.840 on its cohort). We also train it under the R-learner loss with its full head, to measure what unconstrained deep fusion costs in variance.
The key ablation removes images from $\tau$ while keeping them in $\hat m$ and $\hat e$. It isolates the claim that images modify benefit, rather than merely sharpening prognosis or adjusting confounding.
All methods share the same frozen embeddings, cross-fitting folds, overlap region and a fixed hyperparameter budget of 50 configurations, tuned on inner folds. The primary metric is no reported result found on Lacuna for C-for-benefit, a concordance measure computed on matched treated and untreated pairs, or for any benefit-ranking metric on these cohorts. "Beating" a baseline is therefore defined prospectively. Our model must achieve higher C-for-benefit and a higher area under the targeting-operator curve (benefit captured as the treated fraction grows) than risk magnification, with bootstrap 95 percent intervals excluding zero difference on the external gastric sites. It must do so without being worse than the clinical T-learner on calibration of predicted benefit.
Why not just…?
Why not just stratify by a multimodal risk score, as RSA and MRRP do? This is the real threat, and the proposal is designed so that it can win. If treatment reduces hazard by a roughly constant fraction, absolute benefit rises with risk, and a good risk score is already a good benefit score. The gastric and esophageal results, where only the high-risk tier benefits, fit this pattern exactly. The risk-magnification baseline encodes this hypothesis directly. If our estimator does not beat it on external sites, we will report that multimodal data adds prognostic but not effect-modifying information. That is a useful negative result that would validate current cutpoint practice.
Why not just add stage-by-treatment interaction terms to a clinical nomogram? On cohorts with about 50 events, this may well match or beat anything with images, as Wu et al.'s stage-beats-fusion result suggests. Our model contains exactly this as $\tau_{\text{clin}}$ and shrinks the image terms toward zero. It can only diverge from the nomogram when the data support it. The clinical T-learner baseline and the images-removed-from-$\tau$ ablation measure the incremental value directly.
Why not just use a powerful deep fusion network, such as cross-attention or Mamba, end to end on benefit? Benefit is a difference between two arms, so its variance is larger than prognosis, and prognosis already overfits at this scale. In the Wu et al. lung study, learned stacking lost to unweighted averaging. In its kidney cohort, fusion fell to a C-index of 0.617. We include the unconstrained MRRP-style head under the same loss precisely to show this cost. If it wins on the largest cohort, the 1,580-patient gastric set, we report a data-size threshold rather than dismiss it.
Decisive Checks
Three results would settle the core claim. In the semi-synthetic TCGA benchmark, the estimator must recover image-driven benefit when it is planted. In the risk-only regime, it must assign near-zero weight $\gamma_k$ to images. Failure here invalidates any real-data claim. On the external gastric validation sites, the estimator must beat the risk-magnification baseline on C-for-benefit and on policy value at matched treatment rates. Finally, the patients whom the two models reclassify must show the predicted pattern of observed survival within the overlap region. These are patients the risk score would treat but our model would spare, and the reverse. The reclassified groups must also show survival curves that diverge in the predicted direction under propensity weighting, and the divergence must persist under the unmeasured-confounding sensitivity bound. If images improve prognosis but leave these three checks flat, the conclusion is that today's multimodal oncology models are good at telling us who will relapse and silent on who should be treated.
References
- Interpretable Multimodal Fusion Model Enhances Postoperative Recurrence Prediction in Gastric Cancer
- Multimodal deep learning with routine clinical data for recurrence risk stratification in HR+/HER2- early breast cancer
- Survival risk stratification based on prognosis nomogram to identify patients with esophageal squamous cell carcinoma who may benefit from postoperative adjuvant therapy
- Nomogram-based risk stratification to analyze the value of receiving postoperative adjuvant therapy after neoadjuvant immunochemotherapy for patients with locally advanced esophageal squamous carcinoma
- Integrated circulating tumor DNA-based prognostic algorithm for limited stage small-cell lung cancer under definitive chemoradiotherapy and utility in consolidation immunotherapy benefit prediction
- ME-Mamba: Multi-Expert Mamba with efficient knowledge capture and fusion for multimodal survival analysis
- CA-MLIF: Cross-Attention and Multimodal Low-Rank Interaction Fusion Framework for Tumor Prognostic Prediction
- Structural Prognostic Event Modeling for Multimodal Cancer Survival Analysis
- Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction
- Robustness of Multimodal Foundation-Model Forecasting for Postoperative Cancer Outcomes