Aligning Histology to Within-Spot Residuals, Not Spot Averages

Spatial transcriptomics measures gene expression in small tissue "spots". On the widely used Visium platform a spot is about 55µm across and holds several cells, so each measurement is an average that hides how cells in the same spot differ. The paired H&E histology image, a stained microscope picture of the same tissue, shows every individual nucleus. That makes histology the natural modality for pulling cell-level detail back out of the averaged spots. Yet the most common way of teaching an image model about gene expression backfires. A large benchmark found that contrastive pre-training, meaning training image and expression encoders so that matched pairs land close together in a shared embedding space, improves coarse tasks such as predicting mutations but makes direct gene-expression prediction worse than an image model with no such training (HESCAPE).

This note argues that the failure has a specific cause, and that the cause suggests a fix. Most of what histology and expression share at spot scale is the averaged part: cell-type composition, tissue region, and slide-level batch signal. That is exactly the information the spot measurement already provides. The part of histology that is not redundant with the spot is the within-spot variation, meaning which cell differs from its neighbours and in which direction. Standard alignment rewards the redundant part and throws away the useful one. We propose to pre-train image encoders on single-cell-resolution Xenium data against each cell's residual from its local neighbourhood mean. We then test whether this encoder recovers more within-spot heterogeneity when it is used to break spots down into single cells.

Core Claim

Write $x_i$ for the expression vector of cell $i$, and $\bar{x}_{N_\rho(i)}$ for the mean expression of all cells within radius $\rho$ of it. This gives the split

$$x_i = \bar{x}_{N_\rho(i)} + r_i^{(\rho)},$$

where $r_i^{(\rho)}$ is the cell's residual. When $\rho$ matches the spot scale, the first term is roughly what a spot measures, so it is redundant with the spot. The residual is what spot-level data hides.

The claim has two parts. First, the information histology carries about expression is mostly about $\bar{x}$, and contrastive alignment on raw expression keeps mainly that component. Second, an encoder trained with a contrastive loss in which the image embedding of cell $i$ is paired with an embedding of $r_i^{(\rho)}$ will do better at sub-spot deconvolution, with the gain concentrated on within-spot and within-cell-type variation.

Residualizing has a useful side effect. Batch effects are roughly constant within a slide region, so subtracting the local mean also removes much of the batch confound that HESCAPE found tracks retrieval performance.

This framing borrows the shared-versus-unique vocabulary of the multimodal redundancy literature. There, models split representations into shared and modality-specific subspaces with orthogonality constraints (Du et al.; Caragea et al.), or compress inputs with an information bottleneck (Mai et al.). Those methods learn the split from data under a loss weight that is known to be fragile (RAML). Our setting is different because the redundant component is physically known: it is the aggregate the instrument measures. We therefore define the split by construction rather than by a learned penalty.

Research Program

We first make the claim measurable. Using Xenium slides from HESCAPE and HEST-1k (HEST-1k), we fit frozen pathology foundation-model embeddings (UNI, H-Optimus-0, Gigapath) with linear and small nonlinear probes to predict $\bar{x}$ and $r^{(\rho)}$ separately, across radii $\rho$ from single-cell scale up to beyond Visium scale. We do the same with embeddings after HESCAPE-style CLIP and SigLIP pre-training. The first prediction is that pre-training raises how well $\bar{x}$ can be predicted but lowers how well $r$ can be predicted. If that pattern is absent, the diagnosis is wrong and the method has no motivation, so this is the first go/no-go check.

Second, we train residual-aligned encoders. Each cell's image tile is contrasted against an expression encoder that receives $r_i^{(\rho)}$, which we compute on counts normalized by library size. A variant keeps two heads: one aligned to $\bar{x}$ and one aligned to $r$, with the two heads kept independent. This variant lets us check whether the explicit decomposition adds anything over residual alignment alone.

Third, we plug each encoder into a DeepSpot2Cell-style sub-spot model (DeepSpot2Cell). That model sums per-cell predictions so their total matches the spot count, and it is trained only on spot-level supervision. We use simulated 55µm spots from Xenium, where single-cell ground truth is known, and then real Visium slides.

Evaluation separates three quantities: correlation on raw per-cell expression; correlation on the within-spot residual; and correlation on the within-cell-type residual. The third uses cell-type labels from the Xenium ground truth.

Baselines

The naive reference assigns every cell its spot's mean expression, scaled by nucleus area. It scores zero on the residual metrics by construction, and it anchors the raw-correlation metric so we can see how much of any method's score is simply the average.

The strongest existing method is DeepSpot2Cell with a frozen H-Optimus-0 or UNI encoder, run from the released code (ratschlab/DeepSpot) on its lung, breast and pancreas Xenium splits. The DeepSpot2Cell paper reports only relative gains, not absolute values: a 46% improvement in cell-expression correlation over scstGCN on top-100 genes (lung, out-of-sample), and 46%, 65% and 38% improvements across lung, breast and pancreas. No absolute correlation figure for it was found on Lacuna. HESCAPE likewise reports that the un-pre-trained Gigapath encoder had the highest Pearson correlation on the 5K panel, without a value in the file. We will therefore re-run both ourselves.

Every encoder variant gets the same DeepSpot2Cell head and the same splits. Each pre-training variant gets the same Xenium pre-training pairs, the same number of GPU-hours, and the same hyperparameter search budget, with 20 configurations for each method.

Two further competitors isolate the key ingredient. The first is standard CLIP pre-training on raw expression, as in HESCAPE. The second is a learned shared/private orthogonal split in the style of Du et al., trained on raw expression. Both see identical data and compute.

The central ablation sweeps $\rho$. Setting $\rho = 0$ reduces the method to raw-expression CLIP. A slide-wide $\rho$ removes only the batch mean. Beating the baselines means improving the within-spot and within-type residual correlation on held-out donors and on out-of-distribution organs, with confidence intervals from bootstrapping over donors, while keeping raw correlation no worse than the frozen-encoder DeepSpot2Cell.

Why not just…?

Why not just skip contrastive pre-training and use a frozen foundation model, as HESCAPE's result suggests? This is the most serious threat. The frozen Gigapath encoder already beats aligned encoders on raw expression prediction, and DeepSpot2Cell with frozen encoders is strong. Our question is whether any alignment can add information about the residual that frozen features lack, and the frozen baseline is exactly the comparison that answers it. If residual alignment fails to beat the frozen encoder on residual metrics, the practical recommendation becomes "do not align". The diagnostic analysis from the first stage would still stand as the explanation of HESCAPE's paradox.

Why not just deconvolve cell-type proportions with Cell2location or CARD and assign types to nuclei from morphology? The 50-dataset benchmark shows these reference-based tools lead spatial deconvolution (benchmark). If within-spot variation were mostly cell-type identity, a two-step pipeline would work: estimate proportions, then classify each nucleus. We include this pipeline as a baseline. The within-cell-type residual metric is designed so that it cannot score well on it, which separates "recovering composition" from "recovering heterogeneity within a type".

Why not just learn the shared/unique split with an orthogonality penalty or an information bottleneck? Those methods help in emotion recognition and meme classification: Caragea et al. report a 2.6 F1 drop when orthogonality is removed. But their split is learned, with no guarantee that "shared" means "what the spot measures", and their loss weights are fragile. The Du-style baseline trained under a matched budget tests directly whether the known physical split beats a learned one.

Proposed Experiments

The decisive checks come in order.

The first check is the probing asymmetry. On HEST-1k and HESCAPE Xenium slides, CLIP pre-training should raise how well $\bar{x}$ can be predicted and lower how well $r^{(\rho)}$ can be predicted, relative to the frozen encoders.

The second check is the main comparison. On simulated spots, residual-aligned encoders should beat frozen-encoder DeepSpot2Cell, CLIP, the Du-style split, and the Cell2location-plus-morphology pipeline on within-cell-type residual correlation. The gain should peak when $\rho$ is near the spot radius, which a spot-scale mechanism predicts.

The third check is transfer to real spots. On real Visium slides without single-cell truth, we check whether reconstructed spot sums stay consistent with the measured spots.

A secondary analysis tests whether residualization lowers the batch silhouette score that HESCAPE linked to retrieval performance. If it does, that directly supports the claim that residualization removes batch signal.

A negative outcome on the second check with a positive one on the first would still be informative. It would mean the useful within-spot signal is already present in frozen features, and that aligning to spot averages simply destroys it.