Hospitals often want a survival model for their own patients: a rare tumor subtype, a single-center surgical series, or a new treatment arm. Such cohorts typically have 20 to 150 patients with digitized pathology slides. Pretrained image encoders can turn every slide into thousands of feature vectors. At this sample size, though, a model trained on those vectors mostly memorizes noise. The large public benchmark study of multi-omics survival models found that high-dimensional models often lose to a plain Cox regression on age, sex, and stage. We propose a narrower route. Learn a small set of tissue "morphology types" and their prognostic weights once, on a large source cohort. Then, in the small target cohort, keep the clinical variables in the model unpenalized and estimate only how far each morphology weight should move away from its source value. The size of that move is chosen by the target data.
The two-sentence version is this. A slide becomes a short vector: the fraction of its tissue that falls into each of about 16 to 32 morphology clusters learned on a large cohort. In a small new cohort, we fit a Cox model that keeps the clinical stage and age terms free and pulls the morphology coefficients toward the large cohort's values. The pull is strong when data are scarce and weak when data are plentiful. The aim is a model that degrades gracefully as the cohort shrinks, with no cliff where transfer starts to hurt. A second output is a list of morphology types whose prognostic effect does or does not replicate at a new institution. These are candidate morphological biomarkers.
What exists and where the gap is
Three findings motivate the work. First, automated slide features carry prognostic signal beyond stage. Wulczyn et al. trained a patch CNN with average pooling across 10 TCGA cancers (TCGA is The Cancer Genome Atlas, a large public tumor archive). They reported a 3.7% absolute C-index gain over a stage/age/sex model and significant stratification within Stage II and Stage III. The C-index is the fraction of patient pairs whose predicted risks are ordered the same way as their observed survival. That study used roughly 3,600 development cases.
Second, that signal is fragile when molecular or image features compete with clinical variables in small samples. The multi-omics benchmark covered 18 TCGA cancers and found that naive models on more than 100,000 features often underperformed a clinical Cox model. Only Block Forest beat that model on average, and only marginally. CoxBoost was competitive when clinical variables were forced in without shrinkage. That last observation shapes our design.
Third, transfer from a large source cohort helps tabular survival models, but it has a floor. Transfer Survival Forest moved SEER registry trees to a 728-patient hospital cohort. Below about 40 target patients, it did worse than the source model alone. Transfer Survival Boosting re-weights source patients and handles mismatched features. Both methods work on fewer than a dozen clinical variables, not slide features.
The method we reuse comes from Multimodal Prototyping, or MMP. MMP fits a Gaussian mixture to UNI patch embeddings (UNI is a pretrained pathology image encoder), compressing each slide to 16–32 prototypes. It uses those prototypes as tokens for cross-attention with gene-pathway tokens, trained and tested within TCGA splits, with no external transfer and no evaluation below 100 patients. PONET similarly constrains capacity with pathway-structured sparse networks, again only on standard TCGA splits. The nearest attempt at deep-feature transfer into a tiny cohort is the Vision-Mamba CT study. It pretrained on 271 chemoradiotherapy patients, fine-tuned the whole network on 37 immunotherapy patients, and reached a C-index of 0.73 on 150 held-out patients. It had no clinical anchor and did not test what happens at 30, 20, or 10 patients.
None of these papers combines three things: a fixed, interpretable slide representation shared across institutions; clinical covariates protected from shrinkage; and a target-data-driven amount of transfer whose behavior is mapped across cohort sizes. That combination is what we propose.
Core Claim
We claim the following. Represent each slide by its proportions over source-learned morphology prototypes. Keep clinical covariates unpenalized. Shrink the prototype coefficients toward their source estimates by an amount chosen on the target cohort. The resulting model will, at every target size from 20 to 150 patients, match or beat the better of two extremes: the recalibrated source model and the target-only model. It should also beat fine-tuned attention-based multiple-instance models, which learn which patches matter by training a weighting network over all of a slide's patches, when the target cohort has fewer than about 60 patients. Here is the estimator. For patient $i$ with clinical covariates $c_i$ and prototype-proportion vector $z_i \in [0,1]^K$, the hazard is
$$ h(t \mid c_i, z_i) = h_0(t)\exp(\beta^\top c_i + \gamma^\top z_i), $$
where $h_0$ is an unspecified baseline hazard. The target fit maximizes
$$ \ell_{\text{partial}}(\beta, \gamma) - \frac{1}{2\tau^2}\lVert \gamma - \alpha\,\gamma_{\text{src}} \rVert^2, $$
where $\ell_{\text{partial}}$ is the Cox partial log-likelihood on target patients, $\gamma_{\text{src}}$ is the coefficient vector fit on the source cohort, $\alpha$ is a single free recalibration scale, and $\tau$ controls how far target coefficients may drift. As $\tau \to 0$ the model reduces to "source score plus clinical model." As $\tau \to \infty$ it becomes a target-only model. We choose $\tau$ by cross-validated partial likelihood within the target set.
The expectation of graceful degradation is a heuristic. Shrinking a vector of three or more noisy estimates toward an informed guess lowers expected error compared with either the guess or the raw estimates (James–Stein reasoning). The Cox likelihood is only approximately Gaussian at these sizes, so we treat this as a prediction to test, not a guarantee.
Research Program
We will build three components. The first is a source stage. It extracts UNI patch embeddings, fits a $K$-component Gaussian mixture on source patches, and fits $\gamma_{\text{src}}$ in a Cox model on source patients with clinical covariates. Because target patches are soft-assigned to the same fixed prototypes, every institution produces features in the same coordinates. This addresses the feature-mismatch problem that TSF flags as open.
The second is an adaptation stage that implements the penalized fit above. Because $K \le 32$, the fit is cheap enough to repeat hundreds of times per experiment.
The third is a diagnostic stage that probes the known weak point: stain and scanner shift (Obeid et al.). It compares target prototype-occupancy distributions with the source distributions. It also checks whether the per-prototype target estimates agree in sign with the source estimates. A prototype whose effect replicates across institutions, and stays significant after adjusting for stage, is reported as a candidate morphological biomarker. A prototype that flips sign is reported as site-specific. We will inspect top patches per prototype with a pathologist so that each candidate gets a tissue description, not only a heatmap location.
A secondary arm adds about 50 Hallmark pathway scores, as in MMP, as a second penalized block with its own $\tau$. This tests whether molecular features help once clinical variables are protected. The benchmark study's results suggest they often will not, and we will report a null result here if that is what we find.
Baselines
The naive reference is a Cox model on age, sex, and stage fit on the target subsample. The benchmark study shows this is hard to beat. The multi-omics benchmark reports this only qualitatively, with no global C-index tabulated in the file. We will fit it on exactly the same target draws and test patients as every other method.
The strongest existing representation-learning competitor is an MMP-style model. It uses the same UNI features, is trained on the source cohort, and is then fine-tuned on the target subsample. MMP reports a 5.4% average C-index gain over MCAT and SurvPath across six TCGA cancers. The file gives no absolute per-cohort C-index for MMP under site holdout, so the bar is set by our own run. We will match its tuning budget to ours: the same number of hyperparameter configurations, selected by the same inner cross-validation on target data only.
A second deep competitor is an attention-based multiple-instance model (ABMIL/CLAM, per Obeid et al.), pretrained on source and fine-tuned on target. This reproduces the full fine-tuning strategy of the Vision-Mamba study in pathology. That study reports a test C-index of 0.73 after fine-tuning on 37 patients, though in CT and a different disease, so it is a sanity reference rather than a target.
For transfer methods, we run Transfer Survival Boosting on the same prototype-plus-clinical features. It reports a C-index of 0.677 versus 0.500 for target-only Cox on the MAGGIC Euro cohort. We also run a TSF-style forest transfer on the same features. TSF reports a $C^{td}$ of 0.8297 versus 0.7940 for a plain forest on tabular data, along with the 40-patient floor. Both get identical inputs, so any difference comes from the transfer mechanism and not the representation.
Three ablations isolate the key ingredients. The first lets clinical covariates be penalized like everything else. This tests the anchor and should reproduce the benchmark-study degradation. The second replaces source-learned prototypes with mean-pooled UNI embeddings reduced by PCA to $K$ dimensions. This tests whether the prototype representation matters. The third fixes $\tau$ at each extreme. Beating these baselines means a higher mean Uno C-index with no worse integrated Brier score, paired over at least 200 resampled target draws per size. It also means never trailing the best baseline at any target size by more than its paired 95% interval.
Why not just…?
Why not just use the clinical Cox model? It is the honest default. Wulczyn et al. showed slide features add 3.7% C-index and within-stage separation, but with thousands of cases. Whether any of that survives at 30 patients is exactly what our learning curves measure. If our model ties clinical Cox below some size, we will report that size as the point where slides stop being worth using. That answer is useful to a hospital deciding whether to digitize.
Why not just fine-tune an attention-based slide model end to end, as the Vision-Mamba study did? That approach worked at 37 patients in CT. However, fine-tuning there changes millions of parameters using fewer than 40 event-bearing patients, with no protection for clinical signal. MMP's own gains over uncompressed attention models suggest capacity is already the bottleneck within TCGA. The ABMIL and MMP fine-tuning baselines test this directly. We expect them to be competitive above about 100 patients and unstable below about 40. If they win at small sizes too, our claim fails.
Why not just apply the source model's risk score and recalibrate it on target data? This is the real threat. It is the $\tau \to 0$ end of our estimator, and with one or two parameters it may be very hard to beat at 20 patients. Our method only adds value if institutions differ in which morphologies are prognostic, not just in overall risk level. The fixed-$\tau$ ablation settles this. If recalibration matches adaptive shrinkage at every size, we will report the finding: source morphology weights transfer intact, and the biomarker-replication analysis becomes the main contribution.
Why not just run Transfer Survival Boosting on the prototype features? TSB handles heterogeneous features and re-weights source patients. It is the closest transfer competitor and is included on identical inputs. Tree ensembles tend to need more target events to re-estimate split points, and TSF's 40-patient floor came precisely from noisy re-estimation of split thresholds. A smooth shrinkage penalty should avoid that cliff, and the paired learning curves will show whether it does.
Proposed Experiments
Source and target cohorts come from TCGA. Because TCGA slides come from many contributing hospitals, holding out whole sites creates genuine institutional shift. For each of COADREAD, BRCA, KIRC, and LUAD, we will hold out one or two large sites as the target. The cancers are chosen so that the image signal exists in at least three of them: Wulczyn found significant prognostic value in BRCA, COAD, and KIRC. We train everything source-side on the remaining sites, subsample target training sets of 20, 30, 40, 60, 100, and 150 patients, and test on the rest of the held-out site.
A second, fully external pair transfers from TCGA-GBM/LGG to CPTAC-GBM. There the incompletely-paired prototyping work reports a cross-domain C-index of 64.01%, which gives a reference for how hard the shift is. Metrics are Uno's C-index, integrated Brier score, and 2-year calibration slope. We report calibration because ranking alone can hide miscalibrated risk, as the position paper on C-index argues.
Three results decide the question. The first is whether adaptive shrinkage ever falls below the better of recalibration and target-only fitting, and whether it shows a negative-transfer floor like TSF's. The second is the target size at which any slide-based method stops beating clinical Cox, estimated separately for each cancer. The third is whether prototypes flagged as replicating in one held-out site also replicate in a second held-out site of the same cancer. That last test separates real morphological biomarkers from artifacts of a single cohort.
References
- Deep learning-based survival prediction for multiple cancer types using histopathology images
- Large-scale benchmark study of survival prediction methods using multi-omics data
- Tackling Small Sample Survival Analysis via Transfer Learning: A Study of Colorectal Cancer Prognosis
- Boosting Transfer Learning with Survival Data from Heterogeneous Domains
- Multimodal Prototyping for cancer survival prediction
- Deep Biological Pathway Informed Pathology-Genomic Multimodal Survival Prediction
- Cross-dataset adaptation of voxel-level deep radiomics for predicting survival in inoperable locally advanced NSCLC treated with immunotherapy
- Advancing Histopathology with Deep Learning Under Data Scarcity: A Decade in Review
- Interpretable Multimodal Cancer Prototyping with Whole Slide Images and Incompletely Paired Genomics
- Position: Stop Chasing the C-index when Evaluating Survival Analysis Models