Cancer survival models increasingly combine very different inputs for each patient: tissue slide images, gene expression, and a short clinical table. The deep networks that do this well are trained on a few hundred patients per cancer type. They output a single risk score rather than a survival curve, and they need workarounds when a patient is missing one of the data types. A random survival forest is an ensemble of decision trees that split patients into groups whose survival over time differs, then report an empirical survival curve for each final group. It is non-parametric (it assumes no fixed shape for risk over time), it works well with few patients, and trees have a classical way to handle missing inputs. The catch is that a forest picks split candidates by sampling features uniformly. If you concatenate a thousand image-embedding dimensions with twenty clinical variables, the clinical variables are almost never considered. This is the same "large modality drowns small modality" failure that deep fusion papers describe, showing up inside the tree.
We propose a random survival forest that samples at the level of data type before sampling features. At each split it first draws a modality, so clinical data gets the same share of splits as the slide images regardless of how many dimensions each has. It then proposes splits suited to that modality. For tables these are ordinary single-variable thresholds. For compressed image and gene tokens they are thresholds on prototype proportions (such as "fraction of tissue that looks like morphology type 7") plus sparse random linear projections. When a patient lacks a modality, the patient is routed down both branches, weighted by the training branch sizes, so the prediction becomes a mixture of leaf survival curves. This removes the need for imputation at test time and lets incomplete patients contribute during training. The claim we will test is this: on small TCGA cohorts, this forest matches deep fusion heads in ranking patients, beats them in calibration, and degrades more gracefully when modalities are missing. We also test whether modality-level sampling, rather than mere dimension balancing, is what makes this work.
What exists and what is different
The closest work falls into three groups. The first group applies forests to multimodal medical data, but only after reducing each modality to a few handcrafted numbers. Chen, Ellis and Patel feed amyloid PET (reduced to one Centiloid value), plasma biomarkers, and demographics into an off-the-shelf survival forest for progression from mild cognitive impairment to Alzheimer's disease. Chaddad et al. feed cross-sequence MRI texture features into a classification forest after turning survival into a short-versus-long label. Both motivate us, because tree ensembles do well at these sample sizes. Neither confronts the dimensional imbalance that arises once a modality is a learned embedding rather than one scalar, and neither handles missing modalities.
The second group comprises deep fusion models that supply our input representations and our strongest competitors. MMP compresses slides into 16–32 Gaussian-mixture morphological prototypes and transcriptomes into about 50 pathway prototypes, then fuses them with a neural head trained under the Cox loss. The Cox loss assumes each patient's risk is a constant multiple of a shared baseline over time. PS3 adds pathology-report prototypes. HEALNet keeps a shared latent array that attends to each modality in turn and skips missing ones. ME-Mamba uses linear-cost state-space scans. We reuse MMP's prototypes as inputs, which keeps the representation fixed so the comparison isolates the prediction head and fusion rule. We differ from all four in replacing the neural head with a forest that outputs full survival curves and treats missing data structurally.
The third group connects trees and networks for survival, and it constrains our design. NSOTree maps ReLU networks to oblique trees trained under the Cox loss and reports C-index 0.6594 against 0.6431 for a standard forest on METABRIC. This is evidence that oblique splits help. It inherits proportional hazards, however, and was tested only on tabular data. We keep oblique splits but use random sparse projections with log-rank selection, so leaves remain Kaplan–Meier curves. MultiFIX pursues interpretable CT-plus-clinical survival but fuses linearly with a Cox model. Its orthogonality penalty is a different answer to modality dominance than ours.
Research Program
The forest takes, for each patient, a set of modality blocks. These are the clinical table, the slide prototype block (mixture proportions plus prototype mean embeddings), the pathway prototype block, and optionally report prototypes. Growing a tree works as follows. At each node a modality $m$ is drawn with probability $\pi_m$. The default is uniform over modalities available at that node, and the alternative is tuned by out-of-bag error. Within that modality, $k$ candidate splits are generated: axis-aligned thresholds for tables and prototype proportions, and sparse random projections (3–5 nonzero weights) of prototype embeddings. The candidate with the largest log-rank statistic wins. Patients missing modality $m$ at a node that splits on it are sent to both children with weights equal to the child fractions among patients with that modality observed. Leaves store weighted Nelson–Aalen estimates, and the forest averages them. One property follows directly. A patient with only clinical data receives a prediction that is a proper average over survival curves of complete patients sharing their clinical path. No synthetic image features are ever created.
Three analysis products come from the trees. The first is per-modality split share and log-rank gain, which shows which data type drives stratification in each cancer. The second is frequent cross-modal paths, for example "grade above X, then stromal prototype share above Y." These are directly auditable thresholds of the kind Chen et al. extract from partial dependence. The third is test of whether the share of splits a modality wins tracks its stand-alone value. This bears on the tension between the Alzheimer result, where one blood marker sufficed, and the oncology results, where fusion helped.
Baselines
The do-nothing reference is the Kaplan–Meier curve of the training cohort, which has C-index 0.5 and sets the floor for integrated Brier score. The next reference is a forest or Cox model on the clinical table alone, which tells us how much the other modalities add at all.
The naive tree baseline is an off-the-shelf random survival forest on the plain concatenation of all blocks, with mtry tuned. It shows whether dimensional domination actually hurts forests here. The more dangerous tree baseline is a dimension-balanced forest. Each modality is reduced by PCA to the same number of components as the clinical table, missing modalities are mean-imputed, and a standard forest is grown. This is the main ablation for our key ingredient, because it equalizes dimensions without modality-level sampling or fractional routing.
The strongest existing method is MMP's own fusion head on identical prototypes. We rerun it with the authors' recipe on our folds. The file reports only an average 5.4% C-index gain over MCAT and SurvPath and 0.33 GFLOPs on LUAD, with no reported per-cohort C-index on Lacuna. We therefore use our rerun numbers as the target. HEALNet is the strongest missing-modality competitor. Lacuna reports only that it averages up to 4.5% above prior fusion and stays robust with 50% modality dropout, with no reported per-cohort number. We will also rerun it. On every method, fairness means the same five-fold patient-level splits, the same prototype features (HEALNet also gets raw inputs as an upper reference), and an equal hyperparameter budget of 50 random-search trials selected on inner validation folds.
Beating a baseline means one of two things. Either the forest's mean C-index across cohorts falls within one standard error of MMP while its integrated Brier score is significantly lower in a paired test across folds and cohorts, or its C-index loss under modality dropout is smaller than HEALNet's at 50% dropout. Internal ablations remove modality stratification, oblique projections, and fractional routing (replaced by imputation) one at a time.
Why not just…?
Why not just run an ordinary survival forest on PCA-balanced embeddings? This is the real threat, and it may match us on complete data. The proposal stands only if modality-level sampling and fractional routing add something measurable. That would show up as better behavior under missing modalities and better use of the many TCGA patients who lack one assay. The dimension-balanced ablation above settles the question directly. If it matches us on complete and incomplete data, we will report that the gain comes from balancing alone and that the routing machinery is unnecessary.
Why not just keep MMP's neural head? It may rank patients as well or better. MMP's head outputs a proportional-hazards risk score rather than a curve, though, and offers no thresholds a pathologist can audit. In other small-cohort settings tree ensembles have held their own against deep models: SMTBoost reports C-index 0.766 against 0.730 for a weighted forest on rare cardiovascular deaths in SEER, outperforming DeepHit and DeepSurv. Whether that transfers to prototype inputs is exactly the complete-data comparison we run. Calibration measured by integrated Brier score is where we expect a difference.
Why not just late-fuse unimodal risk scores with a linear Cox model? It is simple and handles missing modalities by dropping terms. It cannot represent interactions such as "this morphology matters only in high-grade tumors." The evidence from HEALNet and CA-MLIF shows interactive fusion beating late fusion. We include late Cox fusion as a baseline so that this is checked on our splits rather than assumed.
Proposed Experiments
The primary benchmark is six TCGA cohorts used by MMP and PS3 (BLCA, BRCA, KIRC, LUAD, UCEC, GBMLGG), with slides, RNA-seq, and clinical tables. Metrics are Uno's C-index, integrated Brier score, and D-calibration. A second setting, RADCURE head-and-neck CT plus clinical data with its 670-patient external test split, tests transfer to radiology. CT embeddings come from a 3D network trained on the training split. The reference there is MultiFIX's reported C-index of 0.838 (vs. 0.614 for TNM staging) from MultiFIX, which we treat as a target rather than something we expect to exceed with frozen features. METABRIC serves as a tabular sanity check against NSOTree's 0.6594.
Three outcomes decide the proposal. The first is complete-data parity in C-index with MMP's head together with lower integrated Brier score. The second is a flatter C-index curve than HEALNet and imputation-based forests as test-time modality dropout rises from 0% to 75%, plus a gain from training on incomplete patients that imputation baselines cannot use. The third is a clear margin over the dimension-balanced forest. If only the first holds, the contribution narrows to "forests are competitive heads for prototypes." If none holds, the forest route to multimodal survival is not worth pursuing at these scales.
References
- Predicting Progression from Mild Cognitive Impairment to Alzheimer’s Disease using Survival Analysis
- Predicting the Gene Status and Survival Outcome of Lower Grade Glioma Patients With Multimodal MRI Features
- Multimodal Prototyping for cancer survival prediction
- PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction
- HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data
- ME-Mamba: Multi-Expert Mamba with efficient knowledge capture and fusion for multimodal survival analysis
- NSOTree: Neural Survival Oblique Tree
- Automated and Interpretable Survival Analysis from Multimodal Data
- Multitask Boosting for Survival Analysis with Competing Risks
- CA-MLIF: Cross-Attention and Multimodal Low-Rank Interaction Fusion Framework for Tumor Prognostic Prediction