Letting Nucleus Counts Anchor Reference-Free Immune Deconvolution

A tissue biopsy profiled by spatial sequencing yields, at each spot of about 55 µm, a single blended readout of RNA from roughly ten to thirty cells. Recovering which immune cells sit in each spot (deconvolution) normally requires a matched single-cell reference atlas. For non-model species, archival or rare diseased tissue, and tumors whose transcripts are absent from the annotation, that atlas usually does not exist. Methods that work without a reference avoid this dependency, but they are poor at telling apart closely related cell types such as CD8 and CD4 T cells or regulatory T cells. The reason is that blind factorization of the expression matrix is underdetermined. We propose to remove much of that underdetermination using the H&E image that is already scanned alongside every spatial slide. A nucleus segmentation model can count, per spot, how many nuclei look like lymphocytes, epithelial cells, stromal cells, and so on. It cannot tell which lymphocytes are which, and that last split is exactly what expression is good at. So we fix the coarse composition from the image and let the reference-free factorization split each morphological class into molecular subtypes.

The second ingredient makes the method work where no gene annotation exists. Instead of counting reads per annotated gene, we count reads per de novo contig: a sequence reassembled from the reads alone, without a reference genome. We then borrow a trick from metagenomic binning. Contigs joined in the assembly graph (the graph of overlapping short subsequences that assemblers traverse) probably come from the same transcript, so they should be assigned to the same cell-type program. The claim is narrow and testable. Histology-derived nucleus counts, used as soft constraints on per-spot composition, close a substantial part of the documented accuracy gap between reference-free and reference-based deconvolution at the subtype level. The same machinery runs unchanged on assembly-derived features, so it applies to tissues with no reference at all.

What exists and where this sits

The closest work is FAST (Flexible analysis of spatial transcriptomics data (FAST): a deconvolution approach). FAST runs reference-free non-negative matrix factorization with a graph-smoothness penalty: spots that are near each other and similar in median gray-scale intensity of their H&E patch are pushed toward similar proportions. FAST motivates this proposal, since it shows image priors help reference-free unmixing (Pearson 0.93 versus 0.82 for STdeconvolve in its simulation). It also marks the boundary we move past. FAST treats the image only as a similarity cue, it reduces morphology to one scalar, it uses a Gaussian-noise loss on count data, and it was not validated on immune subtypes. We use the image differently. Nucleus classes give each spot a composition target, and the expression data are asked only to split within those targets.

The benchmark by Li et al. (A comprehensive benchmarking with practical guidelines for cellular deconvolution of spatial transcriptomics) supplies the motivating gap: reference-free methods trail reference-based ones on closely related subtypes. It also names image-guided deconvolution as open and the platform shift between single-cell references and spatial data as an unsolved failure mode.

Three reference-based lines constrain the design. Tangram (Deep learning and alignment of spatially resolved single-cell transcriptomes with Tangram) already uses per-spot nucleus counts, but only as a density target for placing reference cells. We take its idea that segmentation counts are informative and drop the reference. GNN-C2L (Spatio-relational inductive biases in spatial cell-type deconvolution) adds neighbor message passing to Cell2location and finds that going beyond one hop oversmooths. We therefore keep any spatial smoothing at one hop and let the per-spot image constraint carry the local information. STHELAR (STHELAR, a multi-tissue dataset linking spatial transcriptomics and histology for cell type annotation) trains CellViT on Xenium-labelled H&E and reports nine-class mean panoptic quality of only 0.369. This is the central constraint on our method. Image counts are noisy, especially across fine classes, so they must enter as soft, confusion-calibrated targets over a few coarse classes, not as hard facts.

The vision side comes from CellViT (CellViT: Vision Transformers for precise cell segmentation and classification), which reports mean panoptic quality 0.50 on PanNuke. HEST-1k (HEST-1k: A Dataset For Spatial Transcriptomics and Histology Image Analysis) and SIGMMA (SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome) predict spot-level gene expression from images but never decompose spots into cell types. They are competitors only through the "predict then deconvolve" route discussed below.

The sequence side comes from CCVAE (Metagenomic Binning using Connectivity-constrained Variational Autoencoders). CCVAE clusters contigs into genomes using an assembly-graph "should-link" loss together with co-varying coverage across samples. Our spot-by-contig factorization has the same structure: contigs whose counts co-vary across spots belong to one cell-type program, much as contigs whose depth co-varies across samples belong to one genome. CCVAE also warns that spurious graph edges from shared repeats hurt clustering. We therefore weight edges by k-mer multiplicity, as it does, and test the edge term by ablation. The contigs themselves come from standard transcript assemblers such as rnaSPAdes (Extending rnaSPAdes functionality for hybrid transcriptome assembly) run through the transXpress pipeline (transXpress: a Snakemake pipeline for streamlined de novo transcriptome assembly and annotation).

Core Claim

Let $Y$ be the feature-by-spot count matrix, where features are either genes or de novo contigs. Let $S$ be unknown non-negative cell-type signatures and $W$ unknown non-negative per-spot proportions. Each latent cell type $c$ is pre-assigned to one morphological class $m(c)$ (lymphocyte, epithelial/tumor, stromal/fibroblast, myeloid, endothelial). Let $\hat{p}_{ms}$ be the CellViT-estimated fraction of nuclei of class $m$ in spot $s$, with tolerance $\sigma_{ms}$ derived from the segmenter's confusion matrix on held-out data. Let $A_{ij}$ be the multiplicity-weighted assembly-graph edge between contigs $i$ and $j$. We minimize

$$ \mathcal{L} = D_{\mathrm{KL}}(Y \,\|\, SW) + \lambda \sum_{m,s} \frac{\big(\sum_{c:\, m(c)=m} W_{cs} - \hat{p}_{ms}\big)^2}{\sigma_{ms}^2} + \gamma \sum_{i,j} A_{ij}\,\lVert \tilde{S}_{i\cdot} - \tilde{S}_{j\cdot} \rVert^2 $$

subject to $W \ge 0$ and $\sum_c W_{cs} = 1$, where $\tilde{S}_{i\cdot}$ is contig $i$'s signature row normalized to sum to one. The KL term is the Poisson likelihood for counts. The second term fixes how much of each spot belongs to each morphological class. The third term ties contig fragments of the same transcript together. The number of subtypes per class is chosen by held-out reconstruction.

The hypothesis is that, within a class, the subtype split is far better identified than an unconstrained split over all types. The between-class confounds that cause proportion swapping (for example, tumor expression leaking into immune fractions) are removed by the image. Subtypes are then labelled after fitting by marker contigs or genes, and these labels are never used during fitting.

Research Program

The work proceeds in three stages. The first establishes that the image constraint helps on data with trustworthy ground truth. We build pseudo-spots by aggregating STHELAR Xenium cells into 55 µm hexagonal bins. This gives exact cell-type proportions, a co-registered H&E patch, and expression for the same spot. We run CellViT trained on separate STHELAR slides so that the counts are genuinely predicted. The STHELAR labels themselves come from Tangram mapping, so we report results separately on the coarse classes, which are reliable, and on the immune subtypes, where we re-derive labels from canonical marker co-expression in the Xenium panel and report agreement between the two labelings.

The second stage moves to whole-transcriptome spot data in HEST-1k for cohorts whose raw reads are publicly available. There we compare the gene-count and contig-count versions of the same method on identical spots. One limitation is serious. Spot-capture libraries are 3'-biased, so the assembler will recover 3' fragments rather than full transcripts. We accept this, because our features need only be stable count units, not full-length transcripts. We measure fragmentation directly as the number of contigs per annotated gene.

The third stage treats mouse HEST-1k samples as a pretend non-model organism: no mouse annotation and no mouse atlas. The only external resource allowed is the human atlas, mapped through orthologs. This is the regime the method is built for.

Baselines

The naive reference point is the CellViT counts alone, with every subtype inside a class receiving an equal share of the class fraction. It costs nothing and sets the floor for any subtype claim. Coarse-class accuracy of our method must also be at least as good as these raw counts.

The closest reference-free competitor is FAST. We run it twice: as published with gray-scale intensity, and with its graph Laplacian rebuilt from UNI patch embeddings. Both versions get the same features, the same number of latent types, and the same hyperparameter search budget as our method (a fixed grid of equal size, selected on held-out slides). On its own simulation FAST reports Pearson 0.93 versus 0.82 for STdeconvolve, per FAST. STdeconvolve is included as the plain reference-free factorization.

The strongest existing methods overall are reference-based. Li et al. rank CARD, Cell2location, and Tangram highest (benchmark), but no reported numeric score was found on Lacuna for them on these datasets. We run Cell2location and Tangram with an in-distribution reference as an upper-bound oracle. In the mouse stage we also run them with only the ortholog-mapped human reference, which is the realistic competitor.

Ablations isolate each ingredient. We remove the morphology term, replace $\hat{p}$ with ground-truth nucleus counts to measure the cost of segmentation error, shuffle $\hat{p}$ across spots, remove the assembly-edge term, and replace the KL loss with Frobenius.

Beating the baselines means the following. At the subtype level we must achieve lower Jensen–Shannon divergence and a lower CD4/CD8/Treg swap rate than both FAST variants, with paired per-slide tests. In the mouse regime we must at least match ortholog-mapped Cell2location. Closing at least half the gap to the in-distribution reference oracle would count as the strong result.

Why not just…?

Why not just feed deep image embeddings into FAST's smoothness graph? This is the most natural extension and a real threat. A similarity graph only says that two spots resemble each other. It does not say that a spot is 40% lymphocytes, so it cannot stop a factor from absorbing tumor signal into an immune type. The UNI-embedding FAST baseline tests this directly. If it matches our method on subtype divergence, the composition constraint adds nothing and the proposal collapses to that simpler variant.

Why not just read cell types from the image? H&E cannot separate CD8, CD4, and NK lymphocytes, and STHELAR's fine-class quality of 0.369 shows the difficulty even with expression-derived labels (STHELAR). The naive equal-split baseline measures exactly what image-only analysis can give at the subtype level.

Why not predict expression from the image and deconvolve that? Image-predicted expression from HEST-style models is a regression toward visually predictable genes. The subtype markers we need are precisely the ones morphology cannot see. We include this pipeline, using the best HEST encoder with Cell2location on its predictions, but we expect it to lose at subtype resolution.

Why not map reads to a related species' genes instead of assembling? For close relatives this may work as well as assembly, and the mouse experiment with a human reference measures it. Our method can also use those ortholog counts as features. If ortholog counts match contigs, we report that assembly is only needed when orthology is weak, and the de novo component becomes a regime-specific option rather than a core contribution.

Proposed Experiments

The decisive check is whether the gain comes from the composition constraint. Accuracy should degrade smoothly as $\hat{p}$ is corrupted from ground-truth counts, to CellViT predictions, to shuffled counts. With shuffled counts it should fall to or below unconstrained factorization. If CellViT-quality counts already sit near the shuffled level, image noise is too large and the approach fails.

The second check is regional low-abundance immune populations, scored by AUPRC as in GNN-C2L. Here a per-spot image constraint should help most, because smoothing-based methods blur small aggregates.

The third check is contig-based features. On human data, the contig version should come within a small margin of the gene version on spot proportions. The edge term should reduce the number of contig fragments of one gene split across programs. We measure this split rate against annotation that is withheld from the model, using the same edge-weighting test of noisy-graph sensitivity that CCVAE reports, where it achieves F1 0.903 on STRONG100.

Finally, on the pretend-non-model mouse samples, we report proportion divergence against the mouse-reference oracle for every method. This is the result that determines whether the method is useful where it is most needed.