Many spatial transcriptomics methods make each RNA molecule visible by copying a small circular DNA probe into a long tangle of repeats. This copying step, called rolling circle amplification (RCA), turns one molecule into a bright dot roughly a micron across. That works well when molecules are sparse. In cells that express a gene heavily, though, neighbouring dots grow into each other and form a single "plaque", and standard spot-finding software counts that plaque as one molecule or rejects it. The result is the reverse of the familiar single-cell sparsity problem. Single-cell sequencing loses molecules at the low end, as false zeros. RCA imaging loses them at the high end, by squashing many molecules into one. Because the squashing is worst exactly in the cells that differ most from their neighbours, it hides the cellular variation the assay was built to reveal.
We propose to treat a merged plaque as a sum of an unknown whole number of molecules, rather than as an imaging failure that has to be sharpened away. The method fits a physical image model in which every pixel's brightness comes from an integer number of RCA products of known brightness. This model is coupled to a count-based model of how genes are expressed in each local cell type, and the cell type of a neighbourhood is estimated without drawing cell boundaries. The cell-type model supplies the information that makes unmixing a plaque solvable: which genes, in roughly what proportions, are likely to be inside it. The output is a per-neighbourhood estimate of molecule counts with uncertainty attached. The claim we test is that this estimate removes the compression of high expression that current pipelines introduce, and that it recovers high-expressing subpopulations those pipelines blend into their neighbours.
Core Claim
The research file documents the saturation problem clearly. Single molecule chromogenic in situ hybridization assay for RNA visualization in fixed cells and tissues uses padlock probes and RCA with a brown-dye readout. In that work, lengthening amplification from 30 minutes to 4 hours raised detectable products per cell from 43.38 to 178.4, after which the count plateaued because spots merged. Improved in situ sequencing for high-resolution targeted spatial transcriptomic analysis in tissue sections builds a 40-gene RCA panel decoded over sequencing cycles in colon cancer and mouse brain. It reports that high-abundance transcripts form merged plaques that defeat spot finding.
Existing remedies split into two families that do not communicate. ISTDECO: In Situ Transcriptomics Decoding by Deconvolution models the image stack as codebook times barcode intensity times blur, plus background and Poisson noise, and solves for spot intensities. This separates partially overlapping spots. It has no notion of whole-number molecules and no knowledge of which genes co-occur in a cell, so a fully merged plaque leaves it with an underdetermined continuous mixture. The cellular deconvolution family takes counts as given. Count Bridges enable Modeling and Deconvolving Transcriptomic Data treats an aggregate count as an exact sum of latent integer single-cell counts. The CARD, Cell2location and Tangram methods benchmarked in A comprehensive benchmarking with practical guidelines for cellular deconvolution of spatial transcriptomics work from spot-level count vectors. None of them sees the image.
Our structural observation is that a plaque is to molecules what a Visium spot is to cells: an aggregate whose parts must sum to what was observed. Our methodological claim is that this sum constraint becomes identifiable inside an image only when a whole-number brightness model is combined with a cell-type expression prior. The ablations described below are designed to separate those two ingredients.
Research Program
The image model extends ISTDECO's forward model. Let $Y$ be the multi-round, multi-channel image stack, $D$ the barcode codebook, $G$ the blur kernel of the optics, $b$ the background, and $\xi$ Poisson noise. We replace ISTDECO's continuous intensity map with $X = A \odot N$. Here $N_g(p)$ is a non-negative integer giving the number of RCA products of gene $g$ centred at pixel $p$, and $A_g(p)$ is a per-product brightness drawn from a gene- and cycle-specific distribution. That brightness distribution is learned from isolated, confidently decoded spots in the same section:
$$Y = D\,(A \odot N)\,G + b + \xi .$$
Brightness varies because RCA products vary in length, so a plaque's total intensity constrains its molecule count only loosely. The cell-type prior narrows the estimate. Every pixel neighbourhood gets soft cell-type weights $w_k(p)$, estimated segmentation-free in the style of Cell segmentation-free inference of cell types from in situ transcriptomics data, which smooths decoded spot coordinates into a continuous gene-density field and classifies each pixel. Molecule counts summed over a neighbourhood $\mathcal{B}(p)$ then follow a negative binomial distribution, a count distribution whose spread can exceed that of a Poisson:
$$\sum_{q \in \mathcal{B}(p)} N_g(q) \sim \mathrm{NB}\Big(\sum_k w_k(p)\,\theta_{kg},\ \phi_g\Big).$$
In this expression $\theta_{kg}$ is the mean count of gene $g$ in cell type $k$ and $\phi_g$ is a dispersion parameter, following the count-faithful factorization of A fast and efficient count-based matrix factorization method for detecting cell types from single-cell RNAseq data.
Fitting alternates between three steps. First, the type weights are updated from the current count estimates. Second, $\theta$ and $\phi$ are updated. Third, $N$ is updated by a whole-number search: greedy add, remove and swap moves scored by the joint likelihood, run only within plaque regions, with sampling to obtain uncertainty. Pixels already decoded as isolated spots are fixed. This keeps the cost close to ISTDECO's, which decodes a 2048×2048 field in under 30 seconds on a laptop GPU, and it avoids the step-by-step sampling cost that the Count Bridges authors flag for counts in the thousands.
We deliberately do not use the imaging panel to impute genes that were never measured. That is a separate problem, already addressed by reference mapping and by imputers such as Leveraging data-driven self-consistency for high-fidelity gene expression recovery and Clustering by Denoising: Latent plug-and-play diffusion for single-cell embeddings. Both papers warn that unconstrained recovery can invent structure. Our recovery is constrained by measured photons on every pixel. Sparsity enters here as a statistical lens, through whole-number counts and dropout at both ends, and not as an imputation target.
Baselines
The naive reference point is the standard two-step pipeline: threshold-based spot detection, barcode decoding, and assignment of molecules to cells segmented by Cellpose, as used in the improved in situ sequencing work. It defines how severe the compression is, and every other method has to improve on it.
The strongest existing method is ISTDECO, run with its released settings on the same raw image stacks and with the same codebook and blur estimate. The file reports that ISTDECO reaches Pearson r ≈ 0.98 against reference decoders on public MERFISH and ISS data. The file contains no reported per-density F1 table for it on Lacuna, so we will measure ISTDECO's F1 and count error ourselves on every density bin. For ISTDECO and our method alike, counts are aggregated into the same segmentation-free neighbourhoods, so that differences cannot come from how cells are assigned. Beating ISTDECO means two things. In the top density bins, the slope of estimated against true count must be significantly closer to 1. At the same time, F1 in sparse bins must stay unchanged within noise.
A third baseline is linear intensity calibration. Each plaque's background-subtracted integrated intensity in its decoded barcode channels is divided by the median brightness of an isolated product. This is cheap and physically sensible. We give it the same learned brightness distribution and the same tuning budget, which is a grid search over background and threshold settings on held-out fields.
Two ablations isolate the key ingredients. The first keeps whole-number products but removes the cell-type prior, using a flat prior on $N$. The second keeps the prior but makes counts continuous. If the first ablation matches the full model, the cell-type prior adds nothing and should be dropped.
Why not just…?
Why not just shorten the RCA reaction so dots never merge? The smCISH time course argues against this. At 30 minutes only 43.38 products per cell were visible, against 178.4 at 4 hours, so shorter amplification gives up detection of faint products in exchange for avoiding merging. It moves the dropout from the high end to the low end. It does not remove it. We test this directly in the multi-timepoint experiment below. If a short reaction plus standard calling matches our method's accuracy across all expression levels, the method is unnecessary for that tissue.
Why not just use ISTDECO's deconvolution? ISTDECO separates spots that overlap within a few blur widths. Once a plaque is a featureless blob, though, the image alone cannot tell three bright products of gene A from four products split between A and B. Our flat-prior ablation estimates how much whole-number structure alone recovers. The full model's margin over that ablation measures what the biological prior contributes. If ISTDECO already tracks true counts in the densest bins, the gap described in Core Claim does not exist, and we would report that outcome.
Why not just divide plaque intensity by single-product brightness? This is the most serious threat. For single-gene plaques with low variation in product brightness, it may be nearly as good as our method. It should fail in two situations. One is plaques that mix genes, whose cycle-by-cycle colour pattern matches no single barcode and is therefore discarded or miscalled before any intensity is measured. The other is the regime where brightness variation dominates the estimate. We report results separately for single-gene and mixed plaques. If calibration matches the full model on both, we will recommend calibration and present the full model as unnecessary.
Proposed Experiments
The first benchmark is a superposition test built from real images, so ground truth is known exactly. We take fields from the public ISS dataset used by ISTDECO, and from our own reproductions of the 40-gene colon cancer and mouse brain panels, and select regions where products are sparse and confidently decoded. Shifted copies of these regions are added together in raw pixel space. This creates plaques of controlled density whose true per-gene counts are known while keeping real product shapes, brightness variation and background. Metrics are per-neighbourhood count RMSE, the slope of estimated against true count in each density bin, and F1 for barcode identity, computed for all methods.
The second experiment is an orthogonal real-tissue check. In MCF-7 cells and in breast cancer FFPE sections, a highly expressed gene (ACTB, or HER2 in HER2-high regions) is measured by RCA at 30 minutes, 2 hours and 4 hours of amplification. The same cells are also probed with unamplified smFISH in a separate channel. smCISH reports 142.6 ± 70.05 versus 135.7 ± 71.75 molecules per cell for ACTB, which supports smFISH as a reasonable reference. The decisive outcome is whether our 4-hour estimates continue to track smFISH above the point where naive counts flatten.
The third experiment tests whether recovered counts reveal hidden cellular structure. In colon cancer sections, we ask whether the spatial structure of the SPP1–ITGB1 signalling hubs and the differences between tumour core and invasive margin, both reported in the improved in situ sequencing work, sharpen once SPP1 plaques are counted correctly. We also ask whether HER2-high subpopulations separate more clearly in breast tissue. Fold changes between cell types are compared with an adjacent-section single-cell reference.
As a constraint check, segmentation-free typing must stay within the 0.67–0.88 agreement-with-segmentation range that SSAM reports in sparse regions, so that improvements in dense regions do not come at the cost of errors in sparse ones.
The headline test is the density-binned superposition benchmark, compared against intensity calibration and the flat-prior ablation. If neither comparison shows a margin in the top density bins, the proposal fails.
References
- Single molecule chromogenic in situ hybridization assay for RNA visualization in fixed cells and tissues
- Improved in situ sequencing for high-resolution targeted spatial transcriptomic analysis in tissue sections
- ISTDECO: In Situ Transcriptomics Decoding by Deconvolution
- Count Bridges enable Modeling and Deconvolving Transcriptomic Data
- A comprehensive benchmarking with practical guidelines for cellular deconvolution of spatial transcriptomics
- Cell segmentation-free inference of cell types from in situ transcriptomics data
- A fast and efficient count-based matrix factorization method for detecting cell types from single-cell RNAseq data
- Leveraging data-driven self-consistency for high-fidelity gene expression recovery
- Clustering by Denoising: Latent plug-and-play diffusion for single-cell embeddings