Bulk RNA sequencing of a tumor averages gene expression over cancer cells, immune cells and stroma. Deconvolution methods such as CIBERSORT, a regression that unmixes that average into the proportions of 22 immune cell types, have become a standard way to ask which immune populations matter for prognosis. One of the clearest examples in breast cancer singles out immunosuppressive M2 macrophages. The catch is that this readout needs RNA sequencing, and most hospitals do not run it. They do have stained tissue slides. We propose to use CIBERSORT only at training time, as a teacher that labels TCGA breast tumors with continuous immune-cell compositions. A whole-slide histology model learns to reproduce those compositions. We then test whether the image-derived compositions keep their prognostic meaning at external sites where no RNA was ever measured.
The new part is how we validate, not the regressor. Existing bridges between imaging and deconvolution stop at an internal split with binarized labels. External validation in this setting is also hard to read, because the label pipeline (RNA extraction, sequencing batch, reference signature) shifts between sites along with the images. Our design separates the two sources of failure. Composition accuracy is tested where RNA exists, using held-out tissue-source sites. Prognostic utility is tested where RNA does not exist, using survival in an RNA-free external cohort. Each image-derived cell fraction must also beat a direct image-to-survival model and simple histology immune counts. Otherwise the deconvolution intermediate has not earned its place.
Core Claim
A histology model trained to predict continuous, compositionally modeled CIBERSORT fractions should do two things. It should recover cell-type-specific signal, M2 macrophages and CD8+ T cells in particular, on held-out acquisition sites. It should also produce scores that stratify survival in an external cohort without RNA, and add information beyond a coarse immune score. If the M2 estimate adds nothing over a generic lymphocyte count or an end-to-end survival model, the claim is false. We treat that outcome as a real possibility, not a formality.
Positioning Against Prior Work
The closest work is Qian et al.. That study ran CIBERSORT on TCGA-BRCA, found M2 macrophages independently prognostic (HR = 33.288), and trained LASSO radiomics models on 73 TCIA MRI cases to classify M2 as high or low. Its best model, the peritumoral one, reached a test AUC of 0.752. The study motivates ours and also marks its limits: it had no external cohort, it binarized a continuous target, and it used hand-crafted features from old scanners.
We differ in four ways. We use histology as the primary modality. We predict full continuous compositions. We train on roughly an order of magnitude more patients, the TCGA-BRCA slides with matched RNA. And we make external transport the main test.
MESH-HR supplies most of our method. It combines UNI patch embeddings and gated attention-based multiple-instance learning with an XGBoost genomics branch, and it validates zero-shot from DFCI PROFILE to TCGA-BRCA and a cancer-of-unknown-primary cohort. It also shows that continuous scores beat binary clinical labels for survival discrimination (C-index 0.661 vs 0.646). It never models the tumor immune microenvironment. We reuse its vision branch and its continuous-over-binary lesson, and point them at a transcriptomic target it ignores.
C^3M shows that a model trained with a rich modality can be distilled into one that runs without it at data-poor sites. We borrow that logic, with RNA as the training-only modality and slides as the inference modality. Two smaller precedents bound the design. Nassar et al. quantified tumor-infiltrating lymphocytes from H&E across two independent cohorts but did no reference-based deconvolution. Yao et al. showed multi-parametric MRI radiomics holding up at a second center (external AUC 0.86) for a single marker, Ki-67.
Two lines of work constrain the design. Systematic reviews report that hand-crafted radiomics is brittle across scanners: Bedrikovetski et al. found I² = 73% for radiomics versus 0% for deep learning, and Spohn et al. report scanner-sensitive features. This is why MRI radiomics is only a secondary arm here. Large external drops in other domains are a further warning: Ibragimov et al. report AUC falling from 0.906 to 0.74, and Wang et al. report collapse to 0.50 at a database missing inpatient records. Together these warn that our external numbers may fall sharply. All models are local argues that local recalibration may beat transport altogether, so we test that directly rather than assume it away.
Research Program
The teacher labels come from CIBERSORT with the LM22 reference, run uniformly on TCGA-BRCA RNA-seq. LM22 is a matrix of expression profiles for 22 immune subsets. Running it the same way on every sample avoids mixing label pipelines.
CIBERSORT fractions are relative: they sum to one within the immune compartment. A rise in one cell type therefore looks like a fall in the others. We address this in two steps. First, we pair each relative fraction with ESTIMATE's ImmuneScore, an enrichment score for overall immune content. Second, we model the composition in centered log-ratio coordinates, a standard transform that makes compositional data behave like ordinary vectors. For tumor $i$ with relative fractions $w_i$ over $K$ cell types, the target is
$$z_{ik} = \log(w_{ik}+\epsilon) - \frac{1}{K}\sum_{j}\log(w_{ij}+\epsilon),$$
where $\epsilon$ is a small pseudocount for zero fractions. A separate head regresses the ImmuneScore.
Some CIBERSORT fits are poor. CIBERSORT reports a per-sample goodness-of-fit (correlation and p-value), and we turn it into a weight $r_i$. The training loss is
$$L = \sum_i r_i \left( \lVert \hat z_i - z_i \rVert^2 + \lambda (\hat s_i - s_i)^2 \right),$$
where $\hat z_i$ is the predicted log-ratio vector, $s_i$ the ImmuneScore and $\lambda$ a balance weight.
The slide encoder is the MESH-HR vision branch: frozen UNI patch embeddings and gated attention pooling. We also add an auxiliary attention readout that separates tumor-nest patches from stromal patches. This tests the peritumoral-signal observation of Qian et al. at the histology scale rather than only in MRI.
The secondary MRI arm repeats Qian's 73-patient setting with continuous targets. Its purpose is to check whether the 0.752 AUC survives a change from binary to continuous targets. We report it with the caution its size requires and do not use it as primary evidence.
Baselines
The naive reference point is the training-set mean composition assigned to every tumor. For survival, the naive reference is stage plus receptor status alone. Any image model that cannot beat the mean composition on held-out sites has learned nothing transportable.
The strongest direct competitor for the survival question is an end-to-end survival model on the same UNI-plus-attention encoder. It is trained on the same TCGA slides with no deconvolution intermediate, with the same tuning budget and the same site-held-out folds. MESH-HR is the cited anchor for what this kind of continuous slide model achieves: C-index 0.661 on DFCI PROFILE for receptor-derived scores. To count as beating it, our M2 and CD8 scores must improve external C-index when added to this model's risk score, or must match it while carrying a cell-type interpretation. Matching is a weaker but still useful outcome, and we will label it as such.
The strongest competitor for the composition question is a histology immune-cell-density readout, a lymphocyte/immune patch fraction, in the spirit of the H&E quantification in Nassar et al. Lacuna lists no reported result for H&E-to-CIBERSORT fraction regression on TCGA-BRCA, so this baseline will be run by us on identical slides. For the MRI arm, the reference is Qian et al.'s peritumoral LASSO model (test AUC 0.752), re-run on the same split and also scored against continuous targets.
Three ablations isolate the key ingredients. The first trains on binarized high/low M2 labels, as Qian did, instead of continuous log-ratio targets. The second drops the reliability weights. The third replaces the 22-cell target with ImmuneScore alone, which tests whether cell-type resolution adds anything over a macro score.
Why not just…?
Why not just train the slide model directly on survival and skip CIBERSORT? This is the most serious threat, and it may win on raw C-index. Our bet is that the deconvolution teacher acts as a biologically shaped constraint. Such constraints usually help most at sites unlike the training data, because the composition target cannot absorb site-specific shortcuts as easily as a survival label can. That bet is testable: if the end-to-end model matches ours externally and our cell fractions add nothing to it, we will report that the intermediate gives interpretability only.
Why not just count lymphocytes on H&E? It is cheaper. Qian et al.'s finding that M2 macrophages, not immune content in general, carry the prognostic signal suggests that a single density cannot separate suppressive from cytotoxic populations. The decisive check is whether the predicted M2 fraction stays significant in a Cox model that already contains the lymphocyte density.
Why not just recalibrate locally at each site, as All models are local recommends? Recalibration needs local outcomes, and that is reasonable. It does not need local RNA, so it complements our approach rather than replacing it. We will report both the frozen transported model and a version whose final layer is recalibrated on a small labeled fraction of the external site. If light recalibration recovers most of the external drop, that becomes the recommended deployment mode.
Proposed Experiments and Decisive Checks
Composition accuracy is measured on TCGA-BRCA with leave-tissue-source-site-out folds. Each fold holds out entire contributing institutions, so the held-out slides come from unseen staining and scanning. Metrics are per-cell-type Spearman correlation against CIBERSORT, Aitchison distance (Euclidean distance in log-ratio coordinates) for whole compositions, and calibration of predicted M2 and CD8 fractions.
The first decisive check is the gap between random folds and site-held-out folds. A large gap means the model is reading site artifacts rather than immune cells.
Prognostic transport is measured on DFCI PROFILE, which has slides and survival but, as used in MESH-HR, no RNA. Access requires collaboration. The cancer-of-unknown-primary cohort is a stress test. Metrics are C-index and hazard ratios for the predicted M2 fraction, M2-to-CD8 log-ratio and ImmuneScore, adjusted for stage and receptor status.
The second decisive check is whether the predicted M2 effect keeps its direction. Its direction should match the adverse association Qian found in RNA-based TCGA analysis, and it should remain significant after adding the end-to-end survival score and the lymphocyte density. A sign flip or a null result externally would falsify the central claim.
The third check uses the stroma-versus-nest attention readout. It asks whether the M2 signal comes mainly from stromal regions, which would mirror the peritumoral advantage Qian reported. If that pattern appears, it is evidence that the model is reading the expected biology rather than a confound.
References
- Magnetic resonance imaging-based radiomics was used to evaluate the level of prognosis-related immune cell infiltration in breast cancer tumor microenvironment
- MESH-HR: Multimodal Fusion of Histopathology and Structured Somatic Genomics for Continuous Breast Cancer Receptor Subtyping
- Democratizing Clinical Risk Prediction with Cross-Cohort Cross-Modal Knowledge Transfer
- Integrated molecular and clinical characterization of pulmonary large cell neuroendocrine carcinoma
- A novel radiomics based on multi-parametric magnetic resonance imaging for predicting Ki-67 expression in rectal cancer: a multicenter study
- Artificial intelligence for pre-operative lymph node staging in colorectal cancer: a systematic review and meta-analysis
- Radiomics in prostate cancer imaging for a personalized treatment approach - current aspects of methodology and a systematic review on validated studies
- Developing and validating COVID-19 adverse outcome risk prediction models from a bi-national European cohort of 5594 patients
- Development and validation of a prognostic model predicting symptomatic hemorrhagic transformation in acute ischemic stroke at scale in the OHDSI network
- All models are local: time to replace external validation with recurrent local validation