Is This Gene Really Not Spatial? Bayes Factors Calibrated on Neighboring Tissue Slices

Spatial transcriptomics measures gene expression at thousands of known locations in a tissue section. One of its most common analyses asks which genes are "spatially variable," meaning their expression follows a pattern across the tissue rather than scattering randomly. Current tools answer with a p-value. A small p-value flags a gene. A large p-value says nothing, because it cannot separate a gene that truly has no spatial pattern from a gene measured too sparsely to show one. We propose to replace this one-sided test with Bayesian model comparison. Each gene gets a Bayes factor, the ratio of how well two competing models predicted the observed counts, which can positively support "no spatial pattern" as well as "spatial pattern." When neither model wins, the gene is reported as undetermined.

The obstacle is well known. With vague priors, Bayes factors penalize flexible models arbitrarily. Our fix is specific to this data. Tissue sections are often cut as consecutive slices from the same block, so each gene's prior can be fitted on the neighboring slice before evidence is computed on the target slice. This is a version of the conditional marginal likelihood, and the adjacent slice gives a natural answer to that method's open question of how much data to condition on. We also compare three hypotheses rather than two, separating genes whose pattern merely tracks where cell types sit from genes with spatial structure beyond cell composition. The central empirical bet is that much of the poor cross-slice reproducibility of spatial-gene calls comes from underpowered genes being forced into "hit" or "non-hit." A three-way verdict should expose and remove that discordance.

Core Claim

Single-slice spatial-gene tools reproduce poorly. Integrated Bayesian non-parametric spatial modeling for cross-sample identification of spatially variable genes (NBIMSVG) reports under 50% overlap of calls across consecutive slices. It addresses this by pooling slices through a shared shrinkage prior fitted with variational inference, but it never computes model evidence and never certifies a gene as non-spatial.

We claim three things. First, the disagreement between slices is concentrated in genes whose data are uninformative. Second, a properly calibrated Bayes factor can identify those genes in advance. Third, among genes with decisive verdicts in both slices, agreement should be far higher than among all called genes.

The methodological contribution is the slice-conditioned evidence. For gene $g$ on target slice $B$ with neighbor slice $A$, we define

$$\log \mathrm{BF}^{(g)}_{21} = \log p(y^{(g)}_B \mid y^{(g)}_A, M_2) - \log p(y^{(g)}_B \mid y^{(g)}_A, M_1).$$

Here $y^{(g)}_A$ and $y^{(g)}_B$ are the gene's counts on the two slices, and $M_1$, $M_2$ are competing models. Each model's prior over hyperparameters is first updated on slice $A$, which removes the arbitrary penalty a vague prior imposes. The second paper this builds on, Bayesian Model Selection, the Marginal Likelihood, and Generalization, shows two things. Plain marginal likelihood under diffuse priors can favor the wrong model. Conditioning on part of the data repairs this, but how much data to condition on is left as a tuning choice. In our setting, a biologically matched neighboring slice supplies that conditioning set with no tuning.

Research Program

Every gene is evaluated under three nested models, all using a negative binomial count likelihood with per-spot sequencing depth.

$M_0$ is the flat model. Expression varies only with sequencing depth.

$M_1$ is the composition model. Expression is a linear function of each spot's estimated cell-type proportions. These proportions come from reference-based deconvolution; A comprehensive benchmarking with practical guidelines for cellular deconvolution of spatial transcriptomics identifies Cell2location and CARD as top performers, so we use those. The probabilistic approach of stereoscope serves as a sensitivity check.

$M_2$ adds a residual spatial field on top of composition. The field is a low-rank smooth surface over spot coordinates. We use a B-spline basis, following NBIMSVG's argument against rigid covariance shapes, with a shrinkage prior on its coefficients.

The comparison $M_1$ versus $M_0$ asks whether the gene is a cell-type readout. The comparison $M_2$ versus $M_1$ asks whether the tissue has spatial organization beyond which cells are present. That second question is what biologists usually mean by a spatial niche. Current pipelines treat it in disconnected stages, which NBIMSVG itself names as a limitation.

Evidence is computed with a Laplace approximation, a Gaussian fit around the posterior peak that makes the evidence integral tractable. With low-rank fields this costs well under a second per gene. A Review of Bayesian Hypothesis Testing and Its Practical Implementations surveys Laplace and bridge sampling as the standard approximations. We will check the Laplace values against nested sampling on a random 500-gene subset. Nested sampling slowly explores shrinking regions of the prior to estimate the evidence integral. We will run it in the style of SYSBIONS, which shows that GPU nested sampling is practical for biological models but has not been applied to spatial data. The Laplace approximation is only trusted where the two agree within roughly one unit of log evidence.

Each gene receives a verdict based on the Bayes factor. Above 10 means "spatial beyond composition." Below 1/10 means "no residual spatial structure." Anything in between is "undetermined." These thresholds are calibrated on simulation, not assumed. The symmetric treatment of null and alternative follows the use of Bayes factors to support absent effects in No Evidence of the Ego-Depletion Effect across Task Characteristics and Individual Differences: A Pre-Registered Study. It also follows the critique of p-values in Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses. Both papers motivate the design; neither touches spatial data.

Baselines

Naive reference point. The simplest comparison is a single-slice frequentist spatial test thresholded at a false discovery rate of 0.05. Every gene that is not called is treated as non-spatial, which is what practitioners do in effect. Run on the same DLPFC slice pairs, this establishes how often calls agree across slices. NBIMSVG reports under 50% overlap for such tools. Beating it means the decisive subset of our verdicts reaches substantially higher cross-slice agreement. We set a target of 80% or more, and require that the undetermined bin contain most of the baseline's discordant genes.

Strongest existing method. NBIMSVG is the strongest competitor. It is the only multi-slice Bayesian spatial-gene model in the file, and it reports 142 spatial genes on DLPFC that every single-sample method missed. We will run it on the same slices with its default prior and the same GPU budget. We will threshold its posterior shrinkage statistic in two ways: to declare spatial genes, and, as a naive inverse, to declare non-spatial genes.

No reported result was found on Lacuna for NBIMSVG's cross-slice agreement, calibration, or null-certification accuracy, so we will measure all three ourselves. Beating NBIMSVG requires two results. On simulated genes, our "non-spatial" verdicts must have a lower false-certification rate at matched coverage. On DLPFC, the agreement of decisive calls across held-out slices must be higher.

Key-ingredient ablations. Two ablations isolate the contribution. The first computes evidence under vague priors, with no slice conditioning. If slice conditioning matters, this version should be biased toward $M_1$ and should be less stable across slices. The second drops $M_1$ and compares $M_2$ against $M_0$ directly. This should re-flag layer-marker genes as "spatial," showing that the composition model changes biological conclusions rather than relabeling them. All variants share the same deconvolution output, gene set, and spline basis.

Why not just…?

Why not just use leave-one-slice-out cross-validation? This is the real threat. Slice-conditioned evidence is itself a form of held-out predictive score: the first factor is fitted on slice $A$ and the score is computed on slice $B$. Predictive log-likelihood on held-out slices could therefore reach similar verdicts. Cost is not a strong objection here, since a low-rank negative binomial model fitted per gene is cheap.

We will compute the leave-one-slice-out predictive difference alongside our Bayes factor on every gene. If the two rank genes nearly identically and give equally calibrated three-way verdicts, we will report that honestly. The contribution would then be the three-model design and the calibrated "undetermined" bin rather than the evidence estimator. Our expectation of an advantage rests on two points. Conditional evidence scores the whole posterior predictive sequentially rather than a single plug-in fit. And the Bayes factor has a calibrated scale for accepting the null, which a raw predictive difference lacks. The simulation calibration experiment will settle this.

Why not just compare the variational lower bound or use BIC? The variational lower bound used to train models such as Starfysh sits below the evidence by a gap that depends on the approximating family. That gap differs between a flat model and a spatial-field model, so differences in the bound mix model fit with approximation error. BIC ignores the prior and counts all spline coefficients as full parameters even when shrinkage removes most of them, so it will over-penalize $M_2$. Probing as Quantifying Inductive Bias makes the analogous point that evidence, not parameter counts, penalizes effective capacity. We include both as comparators in the simulation study and expect BIC to under-call weak residual patterns.

Why not just threshold NBIMSVG's posterior shrinkage, or use a spike-and-slab inclusion probability? A spike-and-slab prior mixes an exactly-zero component with a broad one, and its inclusion probability is itself a Bayes factor in disguise. Implemented with exact evidence, it would be a close relative of our method rather than a competitor. The concern is with NBIMSVG's version: posterior shrinkage computed by variational inference is typically overconfident, so a gene with almost no counts can look confidently shrunk to zero. The permuted-gene experiment below tests exactly this. NBIMSVG also has no composition model, so it cannot separate cell-type readouts from residual spatial structure.

Proposed Experiments

Simulation calibration. We simulate genes on real DLPFC spot coordinates with real per-spot sequencing depths. Ground truth is one of three cases: flat, composition-driven using estimated proportions, or composition plus a residual field. We vary field amplitude and mean expression. The measures are calibration of the posterior model probabilities, false-certification rate for "non-spatial," and the share of truly weak genes assigned "undetermined." The decisive check is that false certification stays near zero as expression falls, with the undetermined share rising instead.

Permuted real genes. We shuffle spot labels for real DLPFC genes. This destroys spatial structure but keeps realistic count distributions. Every permuted gene is truly non-spatial, so the fraction certified non-spatial should rise with sequencing depth and should never be negative evidence for the null. The same test is applied to NBIMSVG's shrinkage threshold.

Cross-slice reproducibility on DLPFC. We use the 12 consecutive DLPFC slices, conditioning on one neighbor and evaluating on the next, and measure agreement between verdicts with Cohen's kappa. The headline test compares two quantities. One is agreement on genes decisive in both slices. The other is overall agreement of the naive and NBIMSVG calls. Most of the naive baseline's discordant genes should fall in our undetermined bin.

Biological controls. Known laminar layer markers in DLPFC should mostly resolve as composition-explained ($M_1$), not residual-spatial. For context, spatial clustering on this dataset reaches a median adjusted Rand index of 0.6 in DeepST, which shows that layer structure is strongly recoverable. On the three-slice squamous cell carcinoma data, we report which genes carry residual structure beyond composition and whether those genes replicate across slices.

Sensitivity to deconvolution. We swap Cell2location for CARD or stereoscope to test whether verdicts depend on the deconvolution method. If more than a small fraction of decisive verdicts flip, the plug-in composition model is a weakness. In that case we would propagate uncertainty in the cell-type proportions.