Where the Tumor Ignores Its Own Mutations: Instability-Conditioned Residual Fusion of Histology and Genomics

Tumors that accumulate many mutations tend to attract immune cells. Mutated proteins are displayed on the cell surface as foreign-looking fragments, called neoantigens, and T cells gather to attack them. This correlation is strong enough that deep networks can estimate tumor mutational burden (TMB, the number of mutations per megabase of DNA) from ordinary stained tissue slides. Mostly they do it by detecting lymphocytes. The correlation is also imperfect in ways that matter clinically. Some tumors are heavily mutated yet immunologically "cold." Some are genomically quiet yet "hot." Most patients selected by instability biomarkers still fail to respond durably to checkpoint immunotherapy. We propose to measure that deviation directly. First, predict what a tumor's immune landscape should look like given its measured genomic instability. Second, train a slide model to capture only what departs from that expectation. Third, test whether the departure carries prognostic information that neither the genomics nor a standard slide model provides.

In two sentences: existing histology–genomics fusion models let the slide re-learn the immune signal that mutation counts already explain, so the image adds little. We fix the instability-explained part as a known offset, force the image branch to model the residual (for example, hypermutated tumors whose lymphocytes are held back in the stroma), and ask whether that residual predicts survival.

Core Claim

Our central hypothesis is that the most useful thing a whole-slide image (WSI, a gigapixel scan of a stained tissue section) can add to genomic instability measurements is the spatial discordance between instability and immune response, not a second estimate of instability. Three observations in the file motivate this.

First, MC-TMB predicts TMB from slides alone at 0.818 AUC on TCGA, and its attention maps land on dense lymphocytic infiltration. When TMB has already been sequenced, that prediction is largely redundant.

Second, PORPOISE fuses slides with mutation, copy-number and expression vectors through an outer-product layer. Its attribution analysis puts only about 10% of the fused signal on histology. The slide branch contributes little once molecular data are present.

Third, the biology says the discordant cases are real. Yuan et al. show that mutation and neoantigen burden track immune scores in TCGA colorectal cancer, but only on average. Mao et al. find that high stemness in gastric cancer co-occurs with elevated TMB yet with an immune-excluded, stroma-rich microenvironment. Wang et al. review how stroma and acquired escape mutations blunt checkpoint response despite biomarker enrichment.

The closest fusion systems predict either instability itself or survival from everything at once. M2Fusion predicts binary MSI status (microsatellite instability, a mismatch-repair failure phenotype) from pathology plus CT. SURVPATH, ME-Mamba and CA-MLIF predict survival from pooled modalities. None of these treats the instability–immunity relationship as a known prior to subtract. That is the new ingredient here.

Research Program

The system has three parts, trained in sequence.

Instability features. For each TCGA patient we compute a small vector $s$ of instability measurements from whole-exome data:

  • TMB;
  • MSIsensor score, with PCR duplicates removed, because Johansen et al. show that skipping this step inflates scores in stable tumors by 126%;
  • fraction of the genome altered by copy-number changes;
  • predicted neoantigen count;
  • POLE proofreading-mutation status, because Johansen et al. show POLE drives hypermutation in microsatellite-stable tumors.

Where subclonal estimates are available we add subclone count. Cazier et al. report that subclone count correlates with mutation burden at Spearman r = 0.83 in bladder cancer, so it may partly duplicate TMB. We include it but test its marginal value.

Expected immune landscape. A deliberately simple model $\psi(s)$ (a regularized regression) predicts, from $s$ alone, a slide-measurable immune phenotype $\phi$. The phenotype has three components:

  • lymphocyte density inside tumor epithelium;
  • lymphocyte density in adjacent stroma;
  • a ratio of the two, which distinguishes infiltrated from excluded patterns.

$\phi$ is computed from patch-level tissue and cell-type classification on the WSI. We fit $\psi$ and compute residuals out-of-fold, so no patient's residual uses its own fit. The residual

$$r = \phi(W) - \psi(s)$$

is the patient's immune discordance. Here $W$ is the slide, $\phi(W)$ the observed spatial immune phenotype, and $\psi(s)$ the phenotype expected from instability. A hypermutated tumor with excluded lymphocytes gets a large negative intratumoral residual.

Residual survival fusion. We then train an attention-based slide aggregator $f(W)$ whose output enters a Cox hazard alongside a fixed genomic offset:

$$h(t \mid W, s, c) = h_0(t)\exp\big(g(s, c) + f(W)\big)$$

Here $h_0(t)$ is the baseline hazard, $c$ is clinical stage, age and cancer type, and $g$ is a Cox model fit first on $s$ and $c$ and then frozen. Freezing $g$ means gradient descent cannot reduce the loss by having $f$ rediscover lymphocyte density as a TMB proxy. Its capacity goes to what the genomics leave unexplained. This is the boosting logic of fitting the residual of a strong, cheap model. It also attacks the 90/10 attribution collapse at the source rather than rebalancing gradients after the fact.

We then analyze what $f$ attends to. We test whether its high-risk regions coincide with negative discordance $r$. We also check whether discordant tumors are enriched for the escape biology named in the literature: stromal expansion, high stemness index, and B2M or interferon-pathway mutations.

Baselines

Naive reference. A Cox model on instability features plus stage, $g(s,c)$ alone, is the "do nothing new" reference. It is what a clinic with sequencing already has. Any slide contribution must beat it with a likelihood-ratio test on held-out folds, not just a higher point estimate.

Strongest existing method. On overlapping cohorts this is SURVPATH, which cross-attends slide patches to 331 transcriptomic pathway tokens. It reports an overall C-index of 0.629 across TCGA BRCA, BLCA, STAD, COADREAD and HNSC. PORPOISE reports 0.630 pan-cancer over 14 types, a different cohort mix, so it serves as a second reference rather than a direct comparison. We rerun SURVPATH from its public codebase on our exact splits, with the same patch encoder and the same hyperparameter search budget as our model. We also run a strengthened variant in which our instability features are appended as extra tokens, so SURVPATH is not handicapped by lacking them. Beating it means a higher mean C-index across the same five-fold splits, with a paired bootstrap interval excluding zero, in the instability-rich cohorts (COADREAD, STAD, UCEC) where the hypothesis predicts the largest gain.

Late fusion. A slide-only attention model is trained on survival, and its risk score is entered with $s$ and $c$ in a Cox model. This is the most dangerous baseline and is discussed below.

Ablations isolating the key ingredient. The first is the identical architecture trained jointly end to end without freezing $g$. It tests whether the fixed offset matters or merely the additive form. The second replaces $s$ with within-cancer-type permuted instability features. If the residual still helps, its value comes from generic immune features rather than discordance.

Why not just…?

Why not just add a slide risk score to a Cox model with TMB and MSI? This is a real threat. For linear models, joint fitting and offset fitting reach the same optimum. The difference arises only because a separately trained slide model spends its capacity on the lymphocyte signal it can most easily learn, which is the part TMB already explains. The late-fusion baseline tests this directly. If it matches our model, the offset training contributes nothing and we will report that. The discordance analysis would then stand as a descriptive finding, without an architectural claim attached.

Why not just predict TMB from slides, as MC-TMB does? When TMB is measured, a slide estimate of it is redundant. Its external AUC also drops from 0.818 on TCGA to 0.732 on CPTAC, which shows the morphology-to-mutation mapping transfers imperfectly. Our target is the complement of what MC-TMB predicts. We include an MC-TMB-style predicted-TMB feature as an extra covariate to show it adds nothing once sequenced TMB is present.

Why not compute the discordance from bulk RNA immune scores instead of images? This is also a real threat. Residualizing ESTIMATE or CIBERSORT immune scores on instability features requires no slides, and the correlations reported by Yuan et al. suggest it would capture much of the signal. The one thing bulk RNA cannot see is spatial arrangement: whether lymphocytes infiltrate tumor nests or are held in stroma. We run the RNA-residual model as a baseline. The image claim survives only if the spatial residual adds prognostic information beyond it. We expect this, if anywhere, in excluded-phenotype tumors.

Proposed Experiments

Cohorts. The primary cohorts are TCGA COADREAD, STAD and UCEC. These contain MSI-high, POLE-mutant and microsatellite-stable tumors, and hence the widest spread of instability. BLCA and LUAD are secondary. All have diagnostic slides, whole-exome calls, RNA and overall survival.

External check. CPTAC, the external set used by MC-TMB, tests whether the fitted expectation $\psi$ transfers across centers. Its survival follow-up may be too limited for hazard modeling, so there we evaluate $\psi$ by residual calibration.

Metrics. We report C-index, time-dependent AUC and likelihood-ratio tests of $f$ over $g$. We report calibration of $\psi$ by cancer type. We also report the hazard ratio between concordant and discordant strata within TMB-high patients.

Decisive checks. We have three:

  1. Does immune discordance stratify survival within TMB-high and within MSI-high tumors after adjusting for stage?
  2. Does the image residual beat the RNA residual?
  3. Do POLE-mutant stable tumors, which Johansen et al. flag as a confounder for single-assay classifiers, fall where the instability-conditioned model expects rather than where an MSI-only model would?

Scope limit on immunotherapy. The file lists checkpoint trials (KEYNOTE-024, CheckMate 017/057, IMpower130, PACIFIC) and a blood-TMB analysis (Chen et al.). No reported result on Lacuna pairs slides, instability features and treatment response in any of them. We therefore treat response prediction as a follow-on requiring a partnered cohort and make no claim about it here.