Many spatial transcriptomics machines measure gene expression in small circular "spots" rather than in single cells. Each spot holds a handful to a few dozen cells, and the machine reports only their summed RNA counts. The standard way to recover which cell types sit inside a spot is called deconvolution: explain the summed counts as a weighted mix of reference cell-type profiles. This works poorly when two cell types have nearly the same expression, because their sums look alike. Tissues, however, are not random bags of cells. They assemble hierarchically, like crystals or nacre. Cells form small recurring neighborhoods (a capillary wrapped by a pericyte and an astrocyte endfoot, say), and those neighborhoods tile larger domains such as cortical layers. Cell types that look alike in expression often differ sharply in who their neighbors are.
We propose to learn this two-level "packing grammar" from imaging-based spatial data, where every cell's identity and position are known, and then use it as a prior when unmixing spots from sequencing platforms. The model has two parts. A local motif model describes which small multisets of cell types can physically fit together inside a window of fixed area. A coarse domain model describes how tissue regions extend across neighboring spots. The method is borrowed from materials science. There, a two-level generative model fitted to electron-microscopy images reproduced disordered boundary structure from 1 to 100 nm (A multiscale generative model to understand disorder in domain boundaries). The central bet is narrow and testable. Most of the gain should come from a constraint inside each spot, namely which combinations of cells can coexist in a bounded space, rather than from smoothing estimates across spots. That gain should be largest for the closely related and rare subtypes that spot-level averaging hides.
Core Claim
Every current spatial deconvolution approach treats the contents of a spot as a vector of proportions. In the benchmark of 18 methods across 50 datasets (A comprehensive benchmarking with practical guidelines for cellular deconvolution of spatial transcriptomics), the probabilistic, NMF, graph, deep-learning and optimal-transport families all do this. Some add a prior across spots, typically smoothing proportions between neighbors. None asks whether a given multiset of cells is a plausible physical arrangement inside a confined region of known size.
We claim that a learned prior over within-spot compositions can resolve subtypes that flat mixtures cannot distinguish. The prior is a distribution over cell-type multisets, conditioned on a coarse domain label and on a cap on cell count implied by spot area and cell size. A cap of this kind, set by the physical boundary of the region, is what Steering self-organisation through confinement calls a hard, static, effective confinement. We further claim that a simple diagnostic measured on imaging data predicts where this helps. The diagnostic is how fast the entropy of local compositions grows with window size, compared with an independent-mixture model.
Prior Work and Where This Sits
Three papers supply the method. Count Bridges (Count Bridges enable Modeling and Deconvolving Transcriptomic Data) models integer transcript counts directly. It treats per-cell counts as latent variables in an expectation-maximization loop and forces them to sum exactly to the observed spot. We reuse that exact-sum count likelihood. Count Bridges places no structured prior on which cells co-occur inside a spot, and that is the piece we add. Dan et al. (A multiscale generative model to understand disorder in domain boundaries) supply the architecture. They combine a Markov model over local motifs, learned as "words" and "phrases" from atomic images, with a coarse random-nucleation model for longer scales. They found that the local model alone could not reach mesoscopic scales. That result constrains our design: we keep a separate coarse domain tier rather than hoping local motifs will extrapolate. Their observation that configurational entropy grows only logarithmically with segment length motivates our entropy diagnostic. Araújo et al. (Steering self-organisation through confinement) supply the framing of a bounded region as a constraint that reshapes the probability of different configurations. It offers no algorithm. We turn its idea into a concrete cap on how many and which cells a spot can hold.
Two papers use hierarchy in deconvolution in a different sense, and we contrast with both. SCDC (SCDC: Bulk Gene Expression Deconvolution by Multiple Single-Cell RNA Sequencing References) resolves broad lineages before sub-lineages along a taxonomic tree. POPPY (Cross-modal transfer learning for mapping bulk transcriptomes at cellular level) aligns bulk and single-cell data using distances on the Cell Ontology graph. In both, the hierarchy is a hierarchy of labels. Ours is a hierarchy of physical arrangement, learned from images.
scDEF (Identifying hierarchical cell states and gene signatures with deep exponential families for single-cell transcriptomics) shows that a two-tier latent hierarchy can absorb batch noise at its lower tier. This motivates a secondary question, whether the domain tier also buffers differences between platforms, but scDEF has no spatial component.
DECONbench (DECONbench: a benchmarking platform dedicated to deconvolution methods for tumor heterogeneity quantification) acts as a caution. It found that simple methods and simple averaging often match elaborate pipelines, and that methods are often evaluated only on favorable data. This is why our evaluation is built around simple competitors and around cases where the prior should fail.
Research Program
Learning the local tier. From MERFISH and seqFISH+ sections we cut tissue into windows matched to a target spot size, from about 10 µm up to the 55 µm Visium scale. In each window we record the multiset of cell types and their adjacency graph. Following Dan et al., we mine a vocabulary of recurring small neighborhoods of three to eight cells. We then fit a model in which a window's contents are an assembly of such motifs. The total cell count is capped by window area divided by the empirical cell footprint for each type, which acts as volume exclusion.
Learning the coarse tier. We fit a Potts-like domain field over the spot lattice, a model in which each spot carries a discrete domain label and neighboring spots prefer the same label. Its domain labels are learned jointly with the motif statistics that each domain prefers.
Inference. For spot $s$ with observed counts $y_s$, we infer its cell-type multiset $m_s$ and domain $d_s$ from
$$ p(m_s, d_s \mid y_s) \propto p_{\text{count}}(y_s \mid m_s)\; p_{\text{motif}}(m_s \mid d_s, A)\; p_{\text{domain}}(d_s \mid d_{\mathcal{N}(s)}). $$
Here $p_{\text{count}}$ is the exact-sum count likelihood from Count Bridges, given the reference profiles of the cells in $m_s$. $p_{\text{motif}}$ is the learned local grammar for spot area $A$. $p_{\text{domain}}$ couples $s$ to the domain labels of its neighboring spots $\mathcal{N}(s)$. Inference alternates between Gibbs updates of domain labels and a constrained search over motif assemblies within each spot. The search stays tractable because the count cap keeps $m_s$ small.
The diagnostic. On each imaging dataset we compute $H_{\text{grammar}}(A)$ and $H_{\text{indep}}(A)$. These are the entropies of window compositions under the learned grammar and under an independent multinomial with the same marginal frequencies. Their gap, $\Delta H(A) = H_{\text{indep}}(A) - H_{\text{grammar}}(A)$, measures how much information the prior adds at spot area $A$. If local structure saturates, as the logarithmic entropy growth in Dan et al. suggests, $\Delta H$ should grow with $A$. We predict that the accuracy gain over flat methods tracks $\Delta H$ across tissues and spot sizes. If it does not, the stated mechanism is wrong even if accuracy improves.
Baselines
The naive reference point is non-negative least squares against the single-cell reference profiles. This is a flat mixture with no prior. It anchors how hard each dataset is, and DECONbench shows that flat NMF-style methods are not weak: its best RNA method, the weighted NMF r_WNM, reached a mean MAE of 0.024 on in-silico PDAC mixtures. That number comes from bulk mixtures, not spatial spots, so it does not transfer directly to our setting.
The strongest existing competitors are the top performers in Li et al. CARD is a spatial NMF that smooths proportions between neighboring spots. Cell2location is a probabilistic count model with priors on cell abundance. Tangram is a deep mapping of single cells onto spots. CARD matters most because it is the purest "smooth across spots" competitor, and Li et al. name it the leader when spot counts are small. Li et al. rank these methods but report no global error number. No reported result was found on Lacuna for CARD, Cell2location or Tangram as a single accuracy figure on the seqFISH+/MERFISH simulated-spot tasks, so we will rerun them ourselves.
Count Bridges is the closest method. It has the same likelihood with no structured within-spot prior. Its paper reports outperforming STDeconvolve on spatial data but gives no number, so we rerun it as well.
All methods receive the same single-cell reference, the same simulated spots and the same hyperparameter tuning budget, set on a held-out validation section. Our grammar is trained only on imaging sections from different animals or donors than the test sections. No method sees test-section layouts.
Ablations isolate the key ingredient. The first replaces the motif grammar with an independent multinomial under the same count cap, which tests whether the cap alone does the work. The second replaces it with a learned pairwise co-occurrence prior, a logistic-normal model with a full covariance over cell types. The third keeps motifs but removes the domain tier. The fourth keeps the domain tier but removes motifs.
Beating the field means two things. First, lower per-spot cell-type MAE and higher rare-type recall than CARD and Count Bridges on held-out sections. Second, the subtype-resolution gain must exceed that of the pairwise ablation. Gains confined to abundant types do not count.
Why not just…?
Why not just smooth proportions across neighboring spots, as CARD does? Smoothing assumes that neighboring spots resemble each other. That assumption is weakest exactly where subtypes matter, at thin layer boundaries and sparse infiltrates. Smoothing also does nothing to rule out impossible compositions inside a single spot. The experiment that settles this is to stratify test spots by distance to the nearest domain boundary. If CARD matches us everywhere, including near boundaries, the within-spot prior adds nothing and we will report that.
Why not just learn a pairwise co-occurrence covariance from the imaging data? This is the real threat. A second-order prior is simpler, fast, and may capture most neighbor dependence. Our claim of needing motifs holds only if compositions have higher-order structure, such as three-cell units that occur together far more often than pairwise statistics predict. The pairwise ablation tests this directly, and $\Delta H$ computed for the pairwise model shows how much higher-order structure exists. If the pairwise prior closes most of the gap, we will drop the motif tier and present the result as confinement plus pairwise priors.
Why not just use a graph neural network over spots? Graph-based methods such as DSTG did not reach the top tier in Li et al. A graph over spots operates only between spots and, as usually built, treats all directions alike. It has no representation of what fits inside one spot. We include a graph neural network trained on the same simulated spots as a supplementary baseline, so the comparison is empirical rather than argued.
Why not just add the cell-count cap to Count Bridges? This is the independent-plus-cap ablation. Confinement alone may explain part of any gain, and separating it out is exactly what that ablation is for.
Proposed Experiments and Decisive Checks
The primary benchmark builds simulated Visium-, Slide-seqV2- and Stereo-seq-sized spots from the seqFISH+ and MERFISH datasets in the Li et al. suite. Each section is binned into windows of the target size, and the single-cell counts in each window are summed. This gives exact ground truth for both composition and arrangement. Metrics are per-spot MAE on cell-type proportions, recall and precision for cell types below 2% abundance, and a subtype confusion score for pairs of reference types whose profiles correlate above 0.9.
The first decisive check is the $\Delta H$ prediction. Across tissues and spot sizes, the gain over CARD and Count Bridges should rank-correlate with $\Delta H(A)$. Tissues with weak local structure should show no gain.
The second check is where we expect the prior to hurt. A prior learned on healthy tissue may suppress real cells in unusual places. We test this two ways. In the first, we transplant cells of one type into atypical neighborhoods in simulation and measure recall of these planted cells. In the second, we train the grammar on control tissue and evaluate on perturbed or diseased sections where those are available among the imaging datasets. We will report the rate at which atypical cells are lost alongside the headline accuracy. A tempered prior, with its influence scaled by a weight tuned on validation data, is the planned fix. If no weight preserves planted-cell recall while keeping the subtype gain, the method is not suitable for discovery settings.
The third check is cross-platform. We train on MERFISH mouse brain and apply the method zero-shot to 10X Visium, mirroring the Count Bridges transfer. Real Visium has no cell-level ground truth. We therefore score agreement with held-out layer marker genes, and test whether the domain tier reduces sensitivity to which single-cell reference is used, as scDEF's results would suggest. We will also run the 3-cell-line 6:3:1 mixture from SCDC as a sanity check. That mixture has no spatial structure, so the grammar should neither help nor hurt there. SCDC reports a Pearson correlation of 0.991 on it, and matching that number is the bar.
References
- A multiscale generative model to understand disorder in domain boundaries
- A comprehensive benchmarking with practical guidelines for cellular deconvolution of spatial transcriptomics
- Steering self-organisation through confinement
- Count Bridges enable Modeling and Deconvolving Transcriptomic Data
- SCDC: Bulk Gene Expression Deconvolution by Multiple Single-Cell RNA Sequencing References
- Cross-modal transfer learning for mapping bulk transcriptomes at cellular level
- Identifying hierarchical cell states and gene signatures with deep exponential families for single-cell transcriptomics
- DECONbench: a benchmarking platform dedicated to deconvolution methods for tumor heterogeneity quantification