Gene Expression as a Free Teacher for Where Slide Classifiers Look

Computer models that read whole-slide pathology images are usually trained with one label per slide, such as "tumor" or "normal," because asking a pathologist to outline tumor regions is slow and expensive. These models often get the slide label right while attending to the wrong tissue, for example stroma, pen marks, or scanner artifacts. The usual fix is to buy back some of the outlining work. Many cancer cohorts, however, already contain a second measurement taken from the same tissue: bulk gene expression, meaning the average activity of about 20,000 genes across the sample. We propose to use that measurement as a source of spatial supervision that costs nothing extra. A sparse statistical link between image patches and a small set of genes can tell the classifier which patches look like the biology that the genes report.

The core tool is sparse canonical correlation analysis (SCCA). SCCA finds a weighted combination of image features and a weighted combination of genes that move together across patients, and it forces most weights to zero so that the result names a few genes and a few visual directions. Projecting each patch onto the image side of that link gives every patch a "molecular agreement" score. We use this score as a soft prior on the classifier's attention during training only, so test-time inference still needs nothing but the image. The setting we target is the one where this matters most: fewer than about 100 training slides, where attention is least trustworthy and expert outlines are hardest to justify.

Core Claim

In small weakly labeled cohorts, a patch prior derived from SCCA against paired bulk expression improves tumor localization and out-of-site robustness of attention-based multiple instance learning, which is the standard approach that treats a slide as a bag of patches and learns which patches matter. The prior is worth a measurable number of expert-annotated slides. It works because sparsity restricts the signal to a few genes, and only that restricted signal survives at small $n$. Dense expression-prediction targets carry too many directions to estimate reliably from so few patients.

Relation to Prior Work

Two papers come closest, and each supplies one half of the idea. ImageCCA applied SCCA to autoencoder features of pre-selected TCGA and GTEx patches against about 20,000 genes. It found morphology–ECM gene links ($p \le 3 \times 10^{-5}$) and 509 image QTLs. It never fed the canonical directions back into a localizing classifier. It also reports that the latent axes are hard to interpret and partly confounded by technical factors such as ischemic time. We take that confounding finding as a constraint on our design. Mixed supervision on Camelyon16 motivates the gap. CLAM trained on slide labels alone reaches only 0.215 localization Dice with 0.864 tile specificity, and adding patch labels on 10% of slides raises these to 0.350 and 0.999. That paper fixes attention with human labels; we try to fix it with molecular data that is already collected.

Two further lines of work supply comparison methods rather than motivation. Unimodal weakly supervised localizers learn only from the image and its slide label. Quantile proxy-labeling MIL thresholds patch scores at the top $\alpha$% and bottom $\beta$% and reaches 0.804 pixel AUC on 130 annotated TCGA slides, but drops to 0.679 on out-of-location organs. HAG-MIL picks high-resolution patches by attention without checking whether they reflect disease. On the multimodal side, the COVID-19 SCCA plus cooperative learning study used SCCA on $n=149$ patients for discovery and fusion on tabular radiomics. That study needs the extra modalities at prediction time and never touches localization.

Two papers act as warnings. Limited generalizability of multivariate brain-based dimensions shows that canonical-correlation-style axes often fail to replicate across cohorts. Unveiling COVID-19 from chest X-ray shows that small-cohort models lock onto site-specific shortcuts. Both mean that canonical directions must be checked for stability and for confounding by site before they can be trusted as teachers. For a structured option, we borrow the pathway-group penalty of Structured SCCA, which selected coherent pathways where plain SCCA fragmented them. It may stabilize the gene side at small $n$.

Research Program

Each slide $b$ is split into patches with features $f_{bi}$ from a frozen ImageNet-pretrained ResNet-50, matching the encoder setup of the quantile-MIL baseline. An attention network assigns weights $a_{bi}$ that sum to one per slide. The pooled slide vector $z_b = \sum_i a_{bi} f_{bi}$ feeds a slide classifier.

Every few epochs we fit $\ell_1$-penalized SCCA between the pooled vectors $Z$ and the slide's expression matrix $Y$. We first remove tissue-source site and batch from both views by regression. This yields an image direction $w$ and a sparse gene vector $v$. We keep the canonical component whose gene score $Yv$ best separates the training labels, choosing it on training folds only so that no test information leaks in. Each patch then gets a score $s_{bi} = f_{bi}^\top w$, and training minimizes

$$\mathcal{L} = \mathcal{L}_{\text{cls}} + \lambda \sum_b \mathrm{KL}\big(\mathrm{softmax}(s_{b\cdot}/\tau)\,\|\,a_{b\cdot}\big),$$

where $\mathcal{L}_{\text{cls}}$ is the slide-label cross-entropy, $\lambda$ sets the prior's strength, and $\tau$ sets how peaked the prior is. The procedure alternates between updating attention and refitting SCCA. Refitting matters because the attention pooling decides which patches enter $Z$. A pathologist-free first round can pool all patches uniformly.

The analysis has two parts. First, we measure the stability of the selected genes by bootstrap and across TCGA tissue-source sites. Second, we annotate what the selected genes mean biologically, since a prior that tracks immune or stromal genes would teach the model to localize the wrong thing for tumor segmentation. The exact low-rank solver of On Sparse CCA solved a 19,672 × 2,149 instance to optimality in under a second. We use it to check whether our penalized solutions are near the best possible at the chosen sparsity level, so that conclusions do not depend on a poor local solution.

Baselines

The naive reference is attention MIL trained on slide labels alone, the CLAM-style setup that gives 0.215 Dice on Camelyon16 in the mixed-supervision study. It uses our exact encoder, patches, and training schedule with $\lambda = 0$, so any gain comes from the prior and nothing else.

The strongest existing localizer is quantile proxy-labeling MIL. The file reports 0.804 pixel-wise AUC on its 130 annotated TCGA kidney, lung, and breast slides at $\alpha=\beta=0.2$, and 0.679 on out-of-location slides. We rerun it on the same subsampled training sets and tune $\alpha$ and $\beta$ over the same grid size we use for $\lambda$ and $\tau$. Beating it means a higher pixel AUC at matched training size, measured by paired bootstrap over test slides. At full data we expect no more than parity. The claim targets $n \le 100$ and the out-of-location set.

For slide-level accuracy we compare against HAG-MIL using its public code. It reports 0.921 AUC on TCGA-NSCLC and 0.982 on TCGA-RCC at full data, and we rerun it at each subsample size. On slide AUC we only need to avoid losing; the claimed gains are in localization.

Three ablations isolate the key ingredient. The first, the decisive control, shuffles expression across patients. The second replaces SCCA with dense CCA, which removes sparsity. The third replaces SCCA with an auxiliary head that predicts all genes from $z_b$.

To make the "worth N annotations" claim, we also train mixed supervision with exact patch labels on $k$ slides. These labels are drawn from a held-out portion of the annotated TCGA slides, so that the reported test set is never used for training. The comparison yields an exchange rate: how many annotated slides our free prior matches.

Why not just…?

Why not just annotate 10% of slides, as mixed supervision does? That approach works, and it may beat us outright. The question is what it costs and whether the free signal gets close. The exchange-rate experiment answers this directly. If our prior matches fewer than one or two annotated slides, the idea has little practical value, and we will report it that way.

Why not just predict gene expression from the slide as an auxiliary task? This is the most serious threat, because it uses the same data with less machinery. Our argument is that at $n<100$ a 20,000-output regression mostly fits noise, while sparse SCCA keeps only the few gene–morphology directions that are estimable. ImageCCA's confounding findings suggest that dense signals absorb technical variation. The auxiliary-head ablation settles this. If it matches us at small $n$, sparsity is not the active ingredient and we would drop the claim.

Why not just use a known tumor-content gene signature as the patch prior? For tumor-versus-normal segmentation, a fixed signature may carry most of the useful information, and we run it as a baseline with the same KL loss. SCCA should matter more for subtyping and for tissues without a good off-the-shelf signature. If the fixed signature ties SCCA everywhere, the contribution shrinks to "expression-derived attention priors help," which is still useful but narrower.

Why not just use a smaller network or a CRF cleanup step? The CXR study found that ResNet-18 overfit less than ResNet-50 but still learned site shortcuts. Co-segmentation and scribble supervision found that CRFs trade away recall or reinforce the model's own errors unless something else filters uncertainty. Neither change adds new information about where tumor is, and our prior does. We include a CRF post-processing variant of the naive baseline so that it can be compared directly.

Proposed Experiments

We use TCGA slides that have matched RNA-seq. For localization, the data are kidney, lung, and breast cases from the quantile-MIL benchmark, evaluated on its 130 annotated slides and 100 out-of-location slides. For slide-level subtyping, the data are TCGA-NSCLC and TCGA-RCC. Training sets are subsampled to 25, 50, 100, and the full set, with ten random draws each. Metrics are pixel AUC and Dice for localization, slide AUC for classification, and gene-selection stability measured by bootstrap Jaccard overlap and by leave-one-site-out overlap.

Three outcomes would confirm the idea. The gain over the $\lambda=0$ model should be largest at 25–50 slides and shrink with $n$. It should disappear under shuffled expression. It should exceed both the dense-CCA and the auxiliary-regression variants. A further check is out-of-location pixel AUC above 0.679 at matched training size, which would show the prior is not encoding site or organ.

Three outcomes would weaken or refute it. If a shuffled-expression model also helps, the prior is acting as a generic regularizer rather than carrying molecular information. If the selected genes change completely across tissue-source sites, the teacher is too unstable to trust. If attention concentrates on stroma whenever SCCA selects ECM genes, as ImageCCA's BRCA result suggests could happen, the prior teaches the wrong tissue for tumor segmentation. That last case would also show that the choice of canonical component, not SCCA itself, is the weak point.