When Does Gaussian Theory Predict Transcriptomic Deconvolution?

Deconvolution means unmixing a pooled RNA measurement into the cells that produced it. A spatial transcriptomics spot pools 10 to 50 cells, and a bulk tissue sample pools millions. Unmixing is an inverse problem: recover many unknowns from fewer measurements. For signal recovery of this kind, signal processing has an unusually precise tool. It comes from approximate message passing (AMP), an iterative estimator that alternates a cheap linear step with a denoising step. Its "state evolution" is a one-line scalar recursion that predicts in advance how large the reconstruction error will be at every iteration. The catch is that these predictions are proved only when the measurement matrix is random: i.i.d. Gaussian, or at least "right-rotationally invariant", which means its right singular vectors look like a uniformly random rotation. The noise is also usually treated as Gaussian. Transcriptomic mixing operators satisfy neither condition. They are non-negative, collinear because of shared housekeeping genes, and deterministic. The noise is discrete Poisson-like counting.

This note proposes to find out whether the Gaussian-derived theory still predicts reality on real transcriptomic operators, and to build the estimator that makes the question testable. Concretely, we would build a vector approximate message passing (VAMP) solver with an exact count likelihood and a learned count prior. We would then measure whether its state evolution, computed only from the spectrum of a real single-cell reference and from a scalar denoiser curve, predicts the actual per-sample error of spot-level expression recovery. If it does, the payoff is twofold. Every sample in a clinical cohort gets a free, simulation-free error estimate. And we get a quantitative answer to the question Count Bridges left open: at what sequencing depth is a Gaussian noise approximation good enough?

Core Claim

We make two claims, each testable and each able to fail. The first concerns operator universality: for the pooling operators that arise in spot deconvolution, VAMP state evolution fed with the operator's empirical singular-value spectrum predicts empirical mean squared error within a small relative tolerance. This should hold after one specific correction, removing the dominant non-negative "mean" direction (the housekeeping component, which acts like a rank-one spike) before running the message-passing iterations. It should fail without that correction.

The second concerns a noise crossover: replacing the Gaussian output step with an exact Poisson or negative-binomial output step lowers error at low counts per gene. State evolution predicts the depth at which the two curves meet, and that predicted crossover depth should match the crossover observed empirically.

The measurement model we target is the underdetermined one, not bulk proportion estimation. For each spot $s$ we observe gene counts $$y_s \sim \text{Poisson}\big(A_s x_s\big),$$ where $x_s \in \mathbb{R}^{KG}$ stacks the unknown expression of $K$ cell types over $G$ genes. The pooling operator $A_s = p_s^\top \otimes I_G$ weights each cell type's expression by its estimated share $p_s$ of the spot and sums, so the spot has $KG$ unknowns and only $G$ observations. When spots are coupled through spatial smoothness or shared reference profiles, the stacked operator becomes large, sparse, non-negative and far from rotationally invariant. That is exactly the regime where Gaussian theory is not supposed to apply.

What Is Already Done, and What Is Not

Three prior works supply the method. VAMP and its extension to generalized linear models (Vector approximate message passing for the generalized linear model) replace i.i.d. Gaussianity with right-rotational invariance and stay stable up to condition number $10^6$ on 1-bit sensing. However, its internal linear step still uses Gaussian pseudo-measurements, it was tested only on synthetic problems, and its state evolution was left unproven. Plug-in VAMP (Plug-in Estimation in High-Dimensional Linear Inverse Problems: A Rigorous Analysis) proves state evolution for arbitrary Lipschitz denoisers, including deep networks. That lets us plug in a learned count prior, provided we train it to denoise Gaussian pseudo-noise. The proof still requires Haar-distributed right singular vectors, which transcriptomic operators do not have.

Count Bridges (Count Bridges enable Modeling and Deconvolving Transcriptomic Data) motivates the proposal and is the method we must beat or match. It models counts with a Poisson birth-death bridge, deconvolves spots by EM with sum-constrained sampling, and shows that continuous Gaussian flow models distort count data. Its authors also note that continuous models become faster at high counts, but they give no theory for where that crossover lies, no recovery guarantees, and no analysis of the mixing operator. Our proposal differs from Count Bridges in two ways: it uses a single deterministic iterative solver instead of a generative sampler, and it predicts its own error.

SToFM (SToFM: a Multi-scale Foundation Model for Spatial Transcriptomics) is the representation-learning alternative for multi-scale spatial data. It has no explicit measurement operator and offers no error prediction. We use it only as a competing downstream baseline.

The obvious combination would be "run GLM-VAMP on deconvolution." To our knowledge from the file, it has not been done. On its own it would also be a weak contribution, because nobody would trust its error predictions on non-random operators. The new element here is testing whether those predictions transfer to real operators, together with the specific correction that makes them transfer.

Research Program

The build has three pieces. First, an output denoiser: the scalar posterior of each pooled count $z$ given the observed count $y$ under a Poisson or negative-binomial likelihood, with a Gaussian message from the linear step, computed by one-dimensional quadrature. Second, an input denoiser: a count prior per cell type, learned from a single-cell reference such as Onek1k or MERFISH and trained as a Gaussian denoiser at many noise levels. Its Lipschitz constant is enforced by spectral normalization, so that the plug-in conditions of the Fletcher et al. analysis hold. We would run it in two versions, a simple negative-binomial mixture and a small network. Third, an operator preconditioner: we project out the leading non-negative singular direction of the reference before running the iterations, and solve for that component separately in closed form.

The analysis has two parts. For each real reference we compute its singular-value spectrum and run state evolution. This gives predicted MSE per iteration and per sequencing depth, for both the Gaussian and the Poisson output channels. We then compare those predictions against Monte Carlo runs on pseudo-spots built from held-out real cells. We would also report how far each operator is from rotational invariance, by measuring how concentrated its right singular vectors are on individual genes. The aim is to identify which structural property makes the prediction break.

Baselines

The do-nothing reference is the reference-mean estimate. It assigns each spot's cell types their average reference profile, scaled by the estimated proportions. Any method that fails to beat this on held-out expression recovery has recovered nothing about the specific spot.

The strongest existing method is Count Bridges EM deconvolution. On simulated MERFISH multicellular spots it reports a Jensen-Shannon divergence of 0.231 and an RMSE of 0.110, against 0.288 and 0.177 for STDeconvolve (Count Bridges enable Modeling and Deconvolving Transcriptomic Data). These appear to be proportion-level metrics. For recovery of cell-type-specific expression within each spot, we found no reported result on Lacuna, so we will regenerate this baseline ourselves. We run Count Bridges with its released settings on identical pseudo-spots, give it the same reference cells for training, and give it a GPU-hour budget no smaller than ours. "Beating" it means matching its proportion metrics within noise while achieving lower expression RMSE, or matching its expression RMSE at a fraction of its compute. Either outcome must come with calibrated per-sample error predictions, which Count Bridges cannot provide.

For bulk proportions we also run NNLS on raw counts and SCDC (SCDC: Bulk Gene Expression Deconvolution by Multiple Single-Cell RNA Sequencing References). SCDC reports a Pearson correlation of $r=0.991$ on the 6:3:1 cell-line mixture. We regard it as the ceiling for that task, and we do not expect to improve on it.

Three ablations isolate the proposal's ingredients. Gaussian-output VAMP on log-normalized counts isolates the count likelihood. VAMP without the mean-direction projection isolates the operator correction. Plain AMP, which Fletcher et al. report diverging above condition number 10, isolates the need for VAMP itself.

Why not just…?

Why not just use NNLS or SCDC? For estimating bulk cell-type proportions, we think they are adequate. With roughly ten cell types and thousands of genes, the problem is overdetermined, and SCDC's $r=0.991$ on cell-line mixtures leaves little room. That is why this proposal does not target proportions. Recovering per-cell-type expression inside a spot is underdetermined by a factor of $K$. Least-squares methods have no prior over expression and therefore return only the reference means. Our first baseline, the reference-mean estimate, makes this comparison explicit.

Why not just estimate error by simulating pseudo-spots from the reference? This is the real threat to the error-prediction claim. Simulation is the standard practice and needs no theory. The difference is that simulation gives an average error for a generic spot. State evolution, run inside the solver, adapts to each spot's own proportions and depth. We will test calibration per sample. We take reference cells from a different donor or batch than the spots, to mimic the reference mismatch that SCDC documents. We then compare the rank correlation between predicted and true per-spot error for simulation-based and state-evolution-based estimates. If simulation is equally well calibrated, we will report that state evolution's only remaining advantage is speed and narrow the claim accordingly.

Why not just log-transform and run Gaussian VAMP with a standard denoiser? At high depth this may work fine, and that possibility is part of the hypothesis. Count Bridges shows that continuous models fail on low counts: its Enformer-style continuous baseline has MSE 2.59 against 0.60 for Count Bridges on Onek1k. But it does not say where continuous modeling stops failing. The Gaussian-output ablation, swept over depth, measures the crossover empirically, and state evolution predicts it. Agreement between the two is the result. If log-Gaussian VAMP is never worse at realistic Visium depths, the count likelihood is unnecessary there, and we will say so.

Why not just use a pretrained foundation model? SToFM needs large pretraining corpora and has no measurement model tying its outputs to the observed spot totals. We run it on the same pseudo-spots for proportion and annotation metrics. Its reported gains, such as a zero-shot ARI of 0.953, concern clustering, not recovery error, so it cannot answer the calibration question at all.

Decisive Experiments

The data are pseudo-spots of 10 to 50 cells drawn from MERFISH mouse brain cells and from Onek1k peripheral blood cells, subsampled to sequencing depths that span roughly two orders of magnitude. Ground truth is known exactly for these pseudo-spots. Qualitative transfer is checked on 10x Visium, where the benchmark is agreement with known anatomical cell-type patterns, as in the Count Bridges zero-shot evaluation.

Three checks decide the proposal. The first tests the operator correction. State-evolution-predicted MSE should track empirical MSE across iterations and depths after the mean-direction projection, and it should visibly fail without it. If the prediction fails even with the correction, the operator-universality claim is false for transcriptomic operators, and we will report which structural diagnostic predicts the failure.

The second tests the noise crossover. The predicted depth at which Poisson-output and Gaussian-output VAMP meet should match the empirical crossover to within one depth bin.

The third tests calibration. Per-spot predicted error should rank true per-spot error better than simulation-based estimates under donor-mismatched references, while expression RMSE stays at or below the regenerated Count Bridges baseline.