Transcript Spillover as Inter-Cell Interference in Spatial Transcriptomics

Imaging spatial transcriptomics measures where each individual RNA molecule sits inside a tissue section. To turn those molecules into cells, software has to decide which cell each molecule belongs to. In dense tissue such as tumor, stroma or muscle, cell boundaries blur and molecules near the edges get assigned to the wrong neighbor. A platform benchmark across 33 tissues identified this boundary confusion as the main source of transcript misassignment and false cell typing (Systematic benchmarking of imaging spatial transcriptomics platforms in FFPE tissues). Wireless engineers know this problem under another name. A phone at the edge of its serving cell receives the wanted signal mixed with "interference" from neighboring base stations, and the edge users are the ones who suffer most. We propose to treat each measured cell the same way: its expression profile is its own signal plus leakage from its neighbors, and the leakage should be estimated jointly with the discrete question of which cell owns each molecule.

The practical stakes are downstream. Biologists summarize changes in gene expression through functional enrichment analysis, which tests whether a known pathway's genes are overrepresented among the genes that change. They also increasingly study cell-to-cell variability, meaning how much genetically similar cells differ from one another. Spillover corrupts both. A T cell pressed against tumor cells picks up tumor transcripts, gains apparent "variability," and can come out enriched for epithelial pathways that it does not run. Our central claim is that a measurable share of both the within-type variability and the pathway enrichment reported from dense iST tissue is neighbor-driven artifact. We further claim that an interference-aware joint assignment model can remove that artifact without also erasing real heterogeneity. Current segmentation-then-analyze pipelines cannot do both.

Core Claim and Model

We model molecule ownership and cell expression together. Let transcript $t$ of gene $g_t$ be observed at position $u_t$. Candidate owner cells $j$ have centers $c_j$ and expression rates $\mu_{j g}$. We use a learned, monotonically decaying distance kernel $K$ and a per-cell size prior $\pi_j$. The ownership probability is

$$P(z_t = j) \propto \pi_j \, K(\lVert u_t - c_j\rVert) \, \mu_{j g_t},$$

where $z_t$ is the discrete owner of transcript $t$. This expression is where the wireless analogy stops being decorative. The expression-rate term plays the role of transmit power, the distance kernel plays the role of path loss, and a transcript between two cells is an edge user whose "association" depends on both. The cell rates $\mu_{j\cdot}$ are drawn from a cell-type-specific distribution with its own variance, not fixed at the type mean. That choice separates our model from shrinkage-style corrections, which would delete real cell-to-cell variability along with the leakage.

The joint problem is mixed discrete–continuous, like the user-association-plus-power problems of wireless resource allocation. It also has the structure that makes those problems tractable. The mobile-edge placement work of Optimizing AI Service Placement and Resource Allocation in Mobile Edge Intelligence Systems proves that fixing the discrete placement leaves a convex continuous subproblem, then alternates between the two. We reuse that method. For a fixed assignment $z$, the Poisson log-likelihood in $\mu$ and the kernel parameters is concave. For fixed rates, assignment separates into independent per-transcript decisions. Each alternating step therefore costs time linear in the number of transcripts, which is why we do not need the branch-and-bound machinery (and its imitation-learning accelerator) of Scalable Network Adaptation for Cloud-Rans. We take one further idea from wireless: max-min fairness. On the Uplink Max-Min SINR of Cell-Free Massive MIMO Systems optimizes the worst-off users and reports 10% outage capacity rather than averages. We report the analogous "boundary outage", which is assignment and profile accuracy on the worst decile of cells ranked by neighbor contact. Optionally, we reweight the likelihood toward those cells.

Research Program

We will build the model above as a post-processing layer that runs on the molecule tables of Xenium, CosMx, MERSCOPE and similar platforms. It can be initialized from any segmentation.

The analytical centerpiece is a spillover diagnostic that needs no ground truth. Within one cell type, we regress each cell's expression and its pathway enrichment scores on the cell-type composition of its immediate neighbors. Correct segmentation should leave an immune cell's epithelial-pathway score independent of how many epithelial neighbors it has. Neighbor dependence that survives correction indicates residual interference. Neighbor dependence that disappears under shrinkage but not under our model indicates that shrinkage was destroying signal.

Prior work plays three roles here. Bering is a molecule-first graph method that classifies transcript pairs as same-cell or different-cell and then clusters them. It is the strongest existing assigner, and it motivates our work by showing that cytoplasmic transcripts matter. It does not model leakage as a signal mixed into neighbors, and it does not evaluate downstream enrichment. The single-cell disease studies that cluster cells, average them and run hypergeometric enrichment motivate the downstream question, because this is the pipeline whose conclusions we test (Analyzing single-cell transcriptome sequencing data reveals immunological mechanisms of microglia-associated diseases). Distributional perturbation models (PRiMeFlow) constrain the design. PRiMeFlow shows that collapsing cells toward means kills real heterogeneity, reporting DEG recall of 0.26 against 0.0073 for the mean-centric CPA baseline on Srivatsan20. That result is why we refuse type-mean shrinkage.

Baselines

The naive reference point is the platform's default segmentation, followed by cluster-mean differential expression and Fisher's exact test on GO and KEGG. This is the pipeline most biology papers actually run.

The first image-based comparison is nuclear-only assignment, which is spillover-free by construction but loses cytoplasm. The second is standard image-based segmentation (Watershed, Cellpose, Baysor).

The strongest competitor is Bering. On its multi-technology benchmark, it reports Adjusted Mutual Information (AMI) improvements of 0.1 to 0.5 over Watershed, Cellpose and Baysor, 20% to 70% more transcripts per cell, and an AMI of 0.795 on 100-µm thick sections (Bering). It is strongest because it already solves assignment at the molecule level.

All methods will receive identical molecule tables. They will pass through the same downstream pipeline: the same cell typing reference, the same differential-expression thresholds, and the same enrichment tests. Each gets an equal hyperparameter search budget. Beating Bering means one of two things. On a ground-truth task, we must match its AMI within noise while improving boundary-outage accuracy. On neighbor dependence of enrichment scores, we must cut it by a margin exceeding bootstrap variation across TMA cores.

No reported result was found on Lacuna for spillover-induced enrichment artifacts under any method, so that number will be established here.

Two ablations isolate the key ingredient. The first is a continuous-only unmixing, which fits each cell's profile as its own profile plus distance-weighted neighbor profiles, with no discrete reassignment of molecules. The second replaces the variance-carrying cell-type distribution with type-mean shrinkage.

Why not just…?

Why not just run Bering and then do enrichment? This is the obvious composition, and it is our main baseline rather than a straw man. Bering's same-cell/different-cell edges are thresholded and clustered. Molecules in ambiguous boundary zones receive hard labels without regard to how strongly each candidate cell expresses that gene. The per-transcript ownership equation uses exactly that information. If Bering already removes the neighbor dependence of enrichment scores, we will report that as the finding and our model's added value will be limited to the boundary-outage metric.

Why not just use nuclear-only transcripts, which cannot leak? Bering documents that this discards 20% to 70% of each cell's transcripts. The loss is uneven: it falls hardest on cytoplasm-rich and small cells, so it trades one bias for another. We include nuclear-only as a baseline. The expected result is low neighbor dependence but poor recovery of cytoplasmic pathways against matched scRNA-seq.

Why not just regress neighbor profiles out of each cell continuously, with no discrete assignment? This is the real threat. A linear unmixing is cheap and might capture most of the artifact. It is ablation one, and the experiments decide it. If continuous unmixing matches the joint model on neighbor dependence and scRNA-seq concordance, the discrete step is unnecessary and we will say so. We expect the joint model to win mainly on boundary-outage cells, where a molecule's gene identity should decide ownership.

Why not just trust cluster-mean enrichment because averaging washes out noise? Averaging does not wash out spillover. Spillover is systematic and grows with neighbor contact, so it survives averaging within any cluster whose members share a neighborhood. The neighbor-regression diagnostic measures this directly on the naive pipeline.

Proposed Experiments

We start with a controlled benchmark that has known truth. We take matched scRNA-seq cells and place them in simulated tissue layouts with realistic density. Molecules are then emitted using the distance kernels observed in each platform, so ownership, true profiles and true within-type variance are all known. The metrics are assignment AMI, profile correlation, boundary-outage accuracy, and recovery of the true within-type variance, where both inflation and deflation count as failures.

Next, on the FFPE tissue microarrays (17 cancer types and 16 normal tissues) with matched 10x FLEX scRNA-seq (Systematic benchmarking of imaging spatial transcriptomics platforms in FFPE tissues), we measure per-cell-type pseudobulk concordance with scRNA-seq. The reference point is CosMx's reported Spearman correlation of 0.77–0.80. We also measure the rate of co-detection of mutually exclusive lineage markers, and the neighbor dependence of GO and KEGG enrichment scores. Serial sections across three platforms let us test whether a spillover artifact is platform-specific. Bering's multi-technology segmentation benchmark supplies the AMI comparison.

The decisive check is one prediction on the TMA data. In immune and stromal cells, the default pipeline's enrichment for epithelial or tumor pathways should rise with the number of tumor neighbors. Our model should flatten that slope while preserving the within-type variance that the simulation says is real, and type-mean shrinkage should flatten it only by also collapsing that variance. If the slope is flat under the naive pipeline to begin with, the central claim is wrong.