Deep models that predict cancer survival from digitized pathology slides share a known weakness. They can tell which hospital a slide came from almost perfectly, and hospitals differ in their patients' baseline risk and in how long they follow patients up. A model trained on pooled multi-center data can therefore score well by learning "this looks like hospital A, and hospital A's patients die sooner" rather than learning tumor biology. This note proposes a small change to the training loss that removes the reward for that shortcut by construction. Patients are compared only against other patients from the same hospital when the model learns who is at higher risk. We pair this loss with a patch-additive model, in which the slide's risk score is an exact sum of per-patch contributions, and use that decomposition to measure how much of each prediction is explained by hospital identity.
The setting is weakly supervised survival analysis on whole-slide images. A slide is treated as a "bag" of thousands of small image patches with one label: the patient's follow-up time, plus a flag for whether death or recurrence was observed. If it was not observed, the record is right-censored, meaning we know only that the event happened after the last contact. The impact of site-specific digital histology signatures on deep learning model accuracy and bias showed that networks identify the submitting institution in The Cancer Genome Atlas (TCGA) with AUROC 0.964–0.998, even after color normalization. Under preserved-site validation, where whole institutions are held out, reported AUROCs dropped by 0.069 on average, and 35.7% of apparently predictable features became undetectable. Survival models are especially exposed to this failure, because both the baseline hazard and the censoring pattern are properties of the hospital.
Core Claim
Standard neural survival models train with the Cox partial likelihood. For each observed event, this loss compares the patient's risk score against everyone still at risk at that time, the "risk set." When risk sets pool patients across hospitals, any feature that identifies a high-hazard hospital earns gradient. Stratifying the risk sets by site changes that. Let $r_i$ be the model's log-risk for patient $i$, $s(i)$ their site, $Y_i$ their observed time and $\delta_i$ their event flag. The stratified loss is
$$\mathcal{L} = -\sum_{s}\sum_{i:\, s(i)=s,\ \delta_i=1}\Big[r_i - \log \sum_{j:\, s(j)=s,\ Y_j \ge Y_i} e^{r_j}\Big].$$
Adding any constant $b_s$ to every $r_i$ within site $s$ leaves $\mathcal{L}$ unchanged. So the component of the risk score that only encodes "which hospital" has zero gradient. Each site also gets its own implicit baseline hazard. Censoring is then only assumed independent of outcome within a site, a weaker condition than the pooled model needs when follow-up practices differ between hospitals.
The claim to be tested has two parts. First, most of the site-shortcut inflation in multi-center survival models from whole-slide images is between-site ordering that this loss removes. Second, it does so without the biology-stripping risk of adversarial domain alignment. Stratification never asks the representation to be site-invariant. It only stops rewarding site-level differences in outcome, so genuine within-site prognostic morphology is untouched even when that morphology is more common at one hospital.
The protection is exact only for components that are constant within a site. Real site signatures vary from slide to slide, so some shortcut signal can survive. That residual is why the audit component described next is needed.
Research Program
The model is a survival version of Additive MIL. That paper moves the final predictor inside the pooling sum, giving $g = \sum_k p(\alpha_k f(h_k))$. The result is patch contributions that equal Shapley values and carry a sign, so a patch can raise or lower the score. That work covered classification only (Camelyon16, RCC, NSCLC). Here the sum is the Cox log-risk, $r = \sum_k c_k$, so each patch adds or subtracts log-hazard. Patch features come from frozen pathology foundation encoders.
Training follows the fixed-length sub-bag mini-batching of nnMIL, with one adaptation: batches are sampled so that each batch contains enough same-site patients to form meaningful within-site risk sets. Because the Cox loss inside a mini-batch already approximates the full risk sets, the batch sampler matters. We will report how sensitive results are to the number of patients per site per batch.
Additivity serves as the measuring instrument. Every slide score decomposes exactly into patch contributions, so we can fit a site classifier to per-patch contributions and to slide scores. This yields the fraction of risk variance explained by site and the specific patches responsible for it, including whether top-contributing patches fall on ink, folds or background. That is a concrete, patch-level shortcut audit, not a heatmap impression. We will also run a discrete-time variant with per-site baseline hazard bins, as in the neural survival models discussed by Loh et al.. In this variant, patch contributions add in the hazard logit, which tests that the finding is not specific to the Cox model.
This design does not buy spatial interaction modeling. Synthetic Data Reveals Generalization Gaps in Correlated Multiple Instance Learning found that even correlated aggregators fall 0.04–0.15 AUROC short of Bayes-optimal on spatial tasks, and an additive model is further from capturing interactions. We therefore treat stratification as the primary contribution, and we test it on a non-additive aggregator as well.
Baselines
The naive reference point is a Cox model whose only input is site identity. On pooled test data, its C-index measures how much ranking can be achieved from hospital alone. Any pooled-evaluation gain that does not beat this number by a clear margin is suspect.
The standard reference is pooled-risk-set Attention MIL (Attention-based Deep Multiple Instance Learning) with a Cox loss, trained on the same features and splits.
The strongest existing competitors are IPGPhormer and MCAT. IPGPhormer is a multi-scale graph transformer that assigns Cox risk to individual patches. It reports a mean C-index of 0.657 on TCGA BRCA, KIRC, LUAD and STAD. MCAT is a genomics-guided co-attention model and reports a C-index of 0.653 over five TCGA cohorts. MCAT requires transcriptomic data, so it serves only as a reference. Neither paper, as summarized in the file, used preserved-site splits, and no reported result was found on Lacuna for any whole-slide survival model under preserved-site cross-validation. We will therefore re-run IPGPhormer and pooled Attention MIL ourselves on identical preserved-site splits. We will first reproduce the random-split number and treat any shortfall as a caveat. Every model gets the same encoder features, the same hyperparameter search budget (equal trial counts on inner site-held-out folds) and the same compute.
Beating the strongest baseline means a higher within-site C-index on held-out institutions and on external NLST data, paired with a lower site-probe AUROC on the risk scores. A higher pooled random-split C-index does not count.
The ablations isolate each ingredient. Stratified versus pooled loss is run on the additive model and on Attention MIL, isolating the loss. Additive versus attention aggregation under the stratified loss isolates the audit architecture's cost. An adversarial site-removal head on the pooled model tests the invariance route.
Why not just…?
Why not just use stain normalization? The site-signature study already tested Macenko and Reinhard normalization and still recovered site identity at AUROC 0.964–0.998 (site signatures), because the fingerprint lives in texture. We include normalized features as a baseline condition, but the evidence says this does not settle the problem.
Why not just evaluate with preserved-site splits and keep the pooled loss? Preserved-site validation measures the shortcut but does not stop the model from learning it. A model that ranks new hospitals by resemblance to high-hazard training hospitals can still look reasonable on pooled external metrics. This alternative is exactly our pooled-loss arm under preserved-site evaluation. If it matches the stratified model on within-site held-out C-index and site-probe AUROC, the proposal fails. The decisive comparison is therefore built in.
Why not just remove site information adversarially, or use domain-linked contrastive training such as FOND? FOND improved domain-linked accuracy by up to +20.3% on VLCS, but it was designed for natural-image classification. Forcing site-invariant representations is risky when prognostic subtypes cluster at referral centers, since it can erase real biology. Stratification makes no invariance demand. We run the adversarial arm. If it matches stratification on within-site C-index without lowering performance on site-enriched subtypes, it is a real competitor, and we will report it as such.
Why not just use Attention MIL with a stratified loss and skip the additive model? This is the most serious threat. We expect it to match on accuracy, because stratification is the active ingredient. The additive model earns its place only if its signed decomposition gives a sharper shortcut audit. We will test that by checking whether it localizes injected artifacts better than attention weights, the advantage Additive MIL showed for tumor localization (AUPRC 0.42 versus 0.36). If it does not, we will recommend stratified Attention MIL.
Proposed Experiments
The main benchmark is TCGA BRCA, KIRC, LUAD, STAD, BLCA, UCEC and GBMLGG. Folds are preserved-site splits produced with the quadratic-programming partitioner of the site-signature study. The primary metric is the site-stratified C-index on held-out institutions. We also report pooled C-index, integrated Brier score with a baseline hazard re-estimated at the held-out site, and site-probe AUROC on risk scores and patch contributions. External transfer runs TCGA-LUAD to NLST, using lung cases from PLCO where available, following the external cohorts in nnMIL and AdvMIL.
The first decisive check is semi-synthetic, so that the ground truth is known. We resample TCGA survival times so that between-site hazard and censoring differences are controlled at several strengths, while within-site biology is kept fixed. The pooled model's site-probe AUROC and held-out within-site C-index should degrade as between-site differences grow. The stratified model should stay flat.
The second check is an artifact-injection test. We add a synthetic site-correlated texture to a subset of slides and ask whether the additive decomposition's positive-contribution patches find it, compared with attention maps.
The third check is a subgroup audit in the spirit of AFISP, which adversarially searches for the patient subgroups where a model performs worst. We search for worst-case subgroups of within-site C-index by stage, age and site, to confirm that stratification does not buy average gains at the cost of site-enriched subtypes.
The proposal is falsified if the stratified loss fails to lower site-probe AUROC, or fails to raise within-site held-out C-index over the pooled arm under preserved-site splits.
References
- The impact of site-specific digital histology signatures on deep learning model accuracy and bias
- Additive MIL: Intrinsically Interpretable Multiple Instance Learning for Pathology
- nnMIL: A generalizable multiple instance learning framework for computational pathology
- Limitations of Binary Classification for Long-Horizon Diagnosis Prediction and Advantages of a Discrete-Time Time-to-Event Approach: Empirical Analysis
- Synthetic Data Reveals Generalization Gaps in Correlated Multiple Instance Learning
- Attention-based Deep Multiple Instance Learning
- IPGPhormer: Interpretable Pathology Graph-Transformer for Survival Analysis
- Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images
- Domain Generalization for Domain-Linked Classes
- AdvMIL: Adversarial multiple instance learning for the survival analysis on whole-slide images
- A data-driven framework for identifying patient subgroups on which an AI/machine learning model may underperform