Spatial transcriptomics measures gene activity at fixed locations across a tissue slice, which a pathologist can also see under a microscope. The most widely used version reads out each location as one "spot" about 55 micrometres wide, so a single spot pools the RNA of several to a dozen cells. The measurement averages them together, and the rare or mixed cell populations that matter most for cancer and immunology disappear into that average. Yet the same slide is stained with hematoxylin and eosin (H&E), and there every nucleus inside the spot is visible as a separate object. This note proposes treating spot unmixing as a retrieval problem. For each nucleus we see, we look up a real, previously profiled cell whose appearance matches it. We then accept only those combinations of looked-up cells whose summed gene counts reproduce what the spot actually measured.
The idea borrows a principle from hybrid image search, "text provides the intent, images provide the identity," and maps it onto biology. The spot's pooled counts say what must be present in aggregate, and the nucleus images say which cell is which. The result is a gene-expression profile assigned to every segmented nucleus. Each profile is a real cell drawn from a reference bank, not an invented one, and the method also reports how much of the spot those cells fail to explain. If it works, ordinary spot-based datasets gain single-cell resolution without new assays. The approach can also be tested decisively on subcellular platforms where the true answer is known.
Core Claim
Each source of evidence alone is insufficient to assign expression to individual nuclei, but together they are much stronger. Image-only lookup of cells is noisy and confounded by technical batch. Pooled-count deconvolution recovers proportions but cannot say which nucleus is which. We claim that a joint score, combining per-nucleus visual similarity with a constraint that the retrieved cells sum to the observed spot counts, will assign the correct cell state to each nucleus more often than either source alone or a deconvolve-then-assign pipeline. We further claim the advantage will be largest exactly where image retrieval is most batch-confounded.
Concretely, consider a spot with segmented nuclei $i = 1, \dots, N$ and observed gene counts $y \in \mathbb{Z}^G$. Let a reference bank contain cells $r$, each with an image embedding $v_r$ and measured counts $x_r$. We choose one reference cell $a_i$ per nucleus to maximize
$$ S(a) = \sum_{i=1}^{N} \mathrm{sim}(v_i, v_{a_i}) \;-\; \lambda \, D_{\mathrm{Pois}}\Big(y \,\Big\|\, \alpha \sum_{i=1}^{N} x_{a_i}\Big), $$
Here $v_i$ is the query nucleus's image embedding, $\mathrm{sim}$ is cosine similarity, $D_{\mathrm{Pois}}$ is Poisson deviance (the natural mismatch measure for integer counts), $\alpha$ is a per-spot capture-efficiency scale fitted in closed form, and $\lambda$ trades visual against molecular evidence. We use Poisson deviance rather than squared error because of the argument in Count Bridges: transcript data are non-negative integers and should be scored as such.
How This Differs From Existing Work
Three lines of prior work each supply part of the mechanism and each stop short of it.
Image-to-expression retrieval already exists. BLEEP aligns H&E patches with Visium spot profiles contrastively, then predicts a new patch's expression as a weighted average of its nearest reference spots. UMPIRE and HESCAPE scale this alignment to large Visium and Xenium collections. In all three, the unit retrieved is a patch or a spot-level mixture, and the spot's own measurement is never used at query time. We retrieve cells, not mixtures, and we use the observed spot as a constraint on the retrieved set.
Deconvolution methods address the mixing problem without using the image. The benchmark by Li et al. covers 18 tools and finds CARD, Cell2location and Tangram strongest. Count Bridges goes further and samples latent single-cell count profiles that sum exactly to the observed aggregate. That is the closest existing idea to ours, but it has no visual grounding, so its generated cells cannot be attached to particular nuclei. POPPY lets a bulk profile query a single-cell atlas through cross-attention, using an ontology-aware distance between cell types. It works entirely in the transcriptomic domain, with no spatial or visual input.
The retrieval literature supplies the search procedure. FINT and XRCE at ImageCLEFphoto 2007 use the text attached to a query's visual neighbours to re-query in a second modality. This is cross-modal pseudo-relevance feedback: treating the top results of a first search as if they were relevant and using them to refine the query. MIRACLE at ImageCLEFphoto 2009 boosts candidates found independently by both modalities. Bolettieri et al. show that weighted score fusion scales to 100 million items. We reuse the feedback loop, with one change: the "second modality" is the spot's unexplained residual counts, not a text field.
Two findings constrain the design. HESCAPE reports a linear relationship between technical batch strength and retrieval failure, so image-only lookup cannot be trusted across donors. The file reports this relationship without a numeric slope. HESCAPE also finds that contrastive pre-training can degrade expression prediction relative to an unaligned vision backbone. We therefore do not pretrain a new joint embedding. We use existing image encoders and place the molecular evidence in the scoring step instead.
Research Program
The reference bank is built from Xenium data in the HESCAPE collection, which covers six gene panels and 54 donors. In this data each segmented cell has an H&E crop and its own transcript counts. Nucleus embeddings come from frozen pathology encoders. HESCAPE reports that UNI, Gigapath and H0mini paired with the DRVI gene encoder gave the highest image-gene recall. We use UNI by default and report Gigapath as a check. A light projection head is trained on reference cells only, so that visual neighbours share cell type.
Query spots are segmented with CellViT, the nucleus segmenter used in HEST-1k, which also supplies the nucleus count $N$ for each spot.
Search proceeds by cross-modal feedback. First, each nucleus retrieves its top-$k$ visual neighbours. Second, the current best assignment is summed and compared with $y$, and the residual counts are computed. Third, every nucleus's candidate list is re-scored by how much swapping in each candidate reduces the deviance. We also pull in new candidates by querying the bank with the residual itself in gene space, which is the feedback step. Finally, the second and third steps repeat until the score stops improving, using coordinate ascent with random restarts.
The output is one reference cell per nucleus and a per-spot residual. A large residual is the method's honest signal that the bank lacks the needed cell state. This answers the reference-completeness weakness that POPPY's attribution module shares.
The analysis has three parts. We measure where morphology versus the count constraint contributes by sweeping $\lambda$. We measure how much the gain depends on per-donor batch strength. We also test whether the residual detects cell types deliberately withheld from the bank.
Baselines
The naive reference point assigns every nucleus in a spot the same profile: the spot counts divided by $N$. Any cell-level gain must beat this, because on homogeneous tissue it is already correct.
The image-only competitor is the method with $\lambda = 0$, which amounts to BLEEP-style nearest-neighbour retrieval moved to cell granularity. It uses the same encoder, the same bank and the same $k$. It also includes an ablation that averages $k$ neighbours instead of choosing one cell.
The constraint-only ablation sets the visual term to zero. It chooses the set of bank cells that best explains $y$ and then assigns them to nuclei at random. This isolates what morphology adds beyond proportions.
The strongest existing pipeline is deconvolve-then-assign. We run Tangram and Cell2location, two of the top three tools in Li et al., on the same spots with the same bank as reference, passing CellViT nucleus counts as priors where the tool accepts them. Inferred types are then matched to nuclei by optimal assignment against the five CellViT classes.
The closest conceptual competitor is Count Bridges EM deconvolution, if its code can be run. It generates per-cell profiles that sum to the spot, which we assign to nuclei in the same way.
Every method receives the same gene panel, the same donor splits, and an equal hyperparameter search budget of 30 configurations, tuned on validation donors. No reported result was found on Lacuna for cell-level accuracy of Tangram, Cell2location or Count Bridges on Xenium pseudo-spots. The file reports qualitative rankings only. We will therefore establish these numbers ourselves. Beating the strongest baseline means higher per-nucleus cell-type accuracy and higher per-nucleus expression correlation on held-out donors, with confidence intervals from donor-level bootstrap that exclude zero.
Why not just…?
Why not just deconvolve with CARD or Cell2location and then match types to nuclei by their CellViT class? This is the real threat. On tissues where cell types are morphologically distinct and present in similar numbers, it may match our method. Its weakness is structural. CellViT's five classes cannot separate, for example, two lymphocyte subsets in one spot, so the assignment step falls back to chance among them. Deconvolution errors from the platform effect documented by Li et al. are then fixed before morphology can correct them. The deconvolve-then-assign baseline above settles the question directly. If it ties us on per-nucleus accuracy within the immune and stromal compartments, the joint scoring is not worth its cost, and we will say so.
Why not just train a strong vision model to predict each nucleus's expression, since HESCAPE found unaligned Gigapath beats aligned models on expression prediction? Image-only prediction is exactly what HESCAPE shows to be batch-confounded, and it ignores a measurement we already have for every spot. We include Gigapath and UNI image-only variants. The decisive evidence is the batch analysis. If the count constraint removes the dependence of accuracy on donor batch strength while image-only methods keep it, the constraint is doing real work.
Why not just condition Count Bridges on nucleus images, as the research file's fourth gap suggests? This is a legitimate and more expressive alternative. It can generate cell states absent from any bank, which retrieval cannot. It also requires training a conditional generative model over integer counts, and its outputs are not real cells. Retrieval is cheaper and auditable, and its residual flags missing states rather than hallucinating them. We treat image-conditioned Count Bridges as follow-up work, and the held-out-cell-type experiment below measures exactly the regime where it should win.
Proposed Experiments
The main benchmark builds pseudo-spots from held-out HESCAPE Xenium donors. Transcripts are binned into 55-micrometre circles on a Visium-like grid. This is the same simulation logic Li et al. apply to seqFISH+ and MERFISH. Every nucleus in a pseudo-spot has a known cell type and known counts. We report the following:
- per-nucleus cell-type accuracy, also scored with a Cell Ontology-weighted error in the spirit of POPPY, so near-miss subtypes are penalized less;
- per-nucleus Pearson correlation on panel genes;
- recall of rare types making up under five percent of cells;
- spot reconstruction deviance.
Splits are by donor and, separately, by gene panel, to probe transfer.
The batch test regresses per-donor accuracy on the silhouette batch score that HESCAPE uses. We predict a flatter slope for our method than for image-only retrieval.
The completeness test withholds one cell type at a time from the bank. It checks whether per-spot residual deviance rises specifically where that type is present, measured as detection AUC.
A real-data check runs on matching-organ Visium samples from HEST-1k, where no cell-level truth exists. There we fit using half the overlapping genes and predict the spot counts of the held-out half from the assigned cells. This tests whether the assignments carry information beyond the genes used to make them.
The proposal fails if constraint-only or deconvolve-then-assign matches it on per-nucleus accuracy. It succeeds if it beats both, with the margin growing on high-batch donors.
References
- Count Bridges enable Modeling and Deconvolving Transcriptomic Data
- Spatially Resolved Gene Expression Prediction from Histology Images via Bi-modal Contrastive Learning
- Towards unified molecule-enhanced pathology image representation learning via integrating spatial transcriptomics
- A Large-Scale Benchmark of Cross-Modal Learning for Histology and Gene Expression in Spatial Transcriptomics
- A comprehensive benchmarking with practical guidelines for cellular deconvolution of spatial transcriptomics
- Cross-modal transfer learning for mapping bulk transcriptomes at cellular level
- FINT: Find Images aNd Text
- XRCE's Participation to ImageCLEFphoto 2007
- MIRACLE (FI) at ImageCLEFphoto 2009
- Searching 100M Images by Content Similarity
- HEST-1k: A Dataset For Spatial Transcriptomics and Histology Image Analysis