Catching Silent Bugs in Spatial Deconvolution with Discriminant-Validity Tests

Spatial transcriptomics measures which genes are active at each point of a tissue slice. The common platforms capture RNA in "spots" that each hold roughly 10 to 50 cells, so software has to estimate which cell types make up each spot. This estimation step is called deconvolution. The tools that do it are increasingly deep models: variational autoencoders that combine gene counts with microscope images, graph neural networks that smooth over neighboring spots, and large probabilistic fits in PyTorch. Like other deep learning code, they can contain bugs that never crash. Examples include a gene list misaligned between the reference and the tissue matrix, a detached gradient that freezes the gene-expression branch, or an off-by-one between spot barcodes and image patches. The pipeline still produces clean, plausible tissue maps. On real tissue there is usually no cell-level ground truth, and the image and neighborhood priors are good at producing believable anatomy, so these bugs can go unnoticed into published biology.

We propose to borrow a specific tool from measurement science. Psychometrics validates a new instrument against a gold standard, and one of its checks is discriminant validity: a measure of trait A must correlate with the gold-standard measure of A (convergent) and must not correlate with the gold-standard measure of unrelated trait B. Our claim is that the most dangerous silent defects in deconvolution are "shortcut" defects, where outputs track spatial position or morphology instead of the measured gene expression. These defects preserve convergent agreement, because layers still look like layers, while breaking discriminant structure. Two families of counterfactual tests make that break visible, and we expect them to catch defects that accuracy on simulated data, training-dynamics monitors, and simple invariance tests all miss. We will test that claim with a benchmark of injected and naturally occurring defects across six deconvolution tools.

What prior work establishes and leaves open

Three results motivate the proposal. When Good and Reproducible Results are a Giant with Feet of Clay audited speech toolkits and found padding bugs in convolution, subsampling, and positional encodings. These bugs gave reproducible, even flattering scores at the default setting. They were exposed only by an inference-batch-size invariance test, where WER on MuST-C v1.0 rose from 10.52 to 54.56. That is direct evidence that a good score can hide broken code. It supplies our metamorphic methodology, but it has no external ground truth and nothing spatial. DeepLocalize and DeepDiagnosis monitor weights, gradients, and activations for symptoms such as dead nodes, frozen weights, or a loss that does not decrease. They report 34/40 detections and 46/53 identifications respectively. They were built for Keras classifiers, exclude generative and graph architectures, and cannot see defects in which training is perfectly healthy but the model fits the wrong signal. Understanding performance problems in deep learning systems and Developing Bug-Free Machine Learning Systems With Formal Mathematics explain why such defects are hard to tell apart from ordinary modeling difficulty. Beyond Accuracy shows that open-source deep learning unit tests mostly check that code runs and that shapes are right.

On the biology side, Li et al.'s benchmark ranks 18 deconvolution tools on 50 datasets, including pseudo-spots built by pooling cells from seqFISH+ and MERFISH. It names CARD, Cell2location, and Tangram as the top performers. It treats every implementation as correct and attributes differences to the models themselves. Starfysh fuses gene counts with H&E image features through a product-of-experts autoencoder and reports RMSE ≈ 0.09. GraphST reports median ARI 0.60 on the DLPFC benchmark. stereoscope validates against Allen Brain Atlas in situ images only by visual inspection. These methods supply the systems under test.

The measurement-validation literature supplies the logic. VCAT shows the full pattern: domain-specific convergent correlations, cross-domain correlations that are significantly weaker, and diagnostic AUC against established references. Validating New Measurement Tools Against Gold Standards frames this as general criterion validation, not something specific to questionnaires. No work we found uses this pattern to decide whether a piece of software is correct.

Core Claim

Take a deconvolution output $\hat{w}_{s,k}$, the estimated proportion of cell type $k$ in spot $s$, and an independent reference measurement $g_{s,j}$ of cell type $j$. The reference can come from a pooled single-cell imaging ground truth, from in situ hybridization, or from a held-out marker panel. Define the validity matrix $R_{kj} = \mathrm{corr}_s(\hat{w}_{s,k}, g_{s,j})$. A correct tool should show a strong diagonal and a weak off-diagonal. We score discriminant structure as

$$D = \overline{R_{kk}} - \overline{|R_{kj}|}_{k \neq j},$$

the mean on-target correlation minus the mean absolute off-target correlation.

Global error metrics mostly reward the diagonal. A tool that paints the correct layered anatomy from morphology or position can keep RMSE and on-target correlation high. It will also create spurious off-target correlations wherever the true cell types share spatial structure, so $D$ falls. We combine this matrix with two counterfactual interventions that act as oracles with known answers. First, expression shuffling: permute gene-count vectors among spots while holding coordinates and image patches fixed. The outputs of a correct tool must follow the counts, so the validity matrix computed against the shuffled ground truth should keep its structure. A tool whose gene branch is dead or masked will instead keep producing the original map. Second, prior shuffling: permute image patches or graph edges while holding counts fixed. For a tool that does not use histology, outputs should barely change. For multimodal tools, the size of the change is the reliance to be reported.

The claim is falsifiable. If defects that damage the molecular signal while leaving spatial plausibility intact are not caught better by these tests than by simulated RMSE and invariance checks, the claim fails.

Research Program

The first deliverable is a defect benchmark for spatial deconvolution. We will wrap six tools that span the paradigms in Li et al.: Starfysh, GraphST, stereoscope, Cell2location, Tangram, and CARD. Each gets defects injected at the level of source patches, drawn from classes documented in the software literature and adapted to spatial data.

The first class breaks the data pipeline. It includes gene order misaligned between the single-cell reference and the spatial matrix, library-size normalization applied twice, cell-type labels permuted within the reference, and a spot index misaligned against the coordinate table. The second class breaks the model. It includes a detached or zero-scaled gradient in the gene-expression expert of the product-of-experts, overdispersion parameters stuck at saturation, and dead units in the encoder. The third class breaks spatial handling, following the padding bugs in Papi et al. It includes image patches mapped to the wrong barcodes by one grid step, zero-padding at tissue edges that bleeds into boundary spots, and graph adjacency that connects spots across a gap or to dummy boundary nodes.

Each defect is run over several random seeds alongside unmodified runs, so the natural run-to-run variability of clean code is measured. We will also search issue trackers and release notes of these tools for fixed silent bugs and re-introduce them where the history allows. This gives a smaller set of real defects to guard against a benchmark built only from bugs we chose ourselves.

The second deliverable is the detector. It computes the validity matrix, $D$, and the two shuffle responses. Each is converted into a z-score against the distribution from clean seeds, and a run is flagged when any score exceeds a threshold calibrated to a 5% false-alarm rate on clean runs. It needs only the tool's inputs and outputs, so it works on any architecture. That is a deliberate contrast with the hook-based monitors, which need framework-specific instrumentation.

Baselines

The naive reference point is the current practice: run the tool with default settings, look at the map, and accept it if the anatomy is plausible. We operationalize this as a check that estimated layer-marker cell types peak in the annotated DLPFC layers. It should catch very little, and it sets the floor.

The strongest existing competitor is accuracy on simulated ground truth, as in Li et al. Pseudo-spots are built by pooling cells from the seqFISH+ and MERFISH datasets, RMSE and Pearson correlation against the known proportions are computed, and a run is flagged if its error exceeds the clean-seed distribution at the same 5% false-alarm rate. This is the strongest competitor because it has exact ground truth and is what the field already trusts. We found no reported per-method RMSE or correlation numbers from Li et al. on Lacuna. The nearest reported figures are Starfysh's RMSE ≈ 0.09 and GraphST's DLPFC median ARI of 0.60. We will first reproduce these on unmodified code as a sanity check on our setup.

The second competitor is training-dynamics monitoring in the style of DeepDiagnosis. We will port its eight symptom checks to PyTorch hooks for the neural tools: numerical errors, unchanged weights, saturated activations, dead nodes, out-of-range values, non-decreasing loss, non-increasing accuracy, and vanishing or exploding gradients. Its thresholds are tuned on the same clean runs. For context, the published results are 46/53 on the SGS benchmark for DeepDiagnosis and 34/40 detection for DeepLocalize, but those numbers come from vision classifiers and do not transfer directly to this setting.

The third competitor is pure invariance testing in the style of Papi et al.. Outputs should be unchanged under spot reordering, minibatch size, tiling of the tissue, and cropping away from the boundary. This competitor uses no external reference.

All detectors see the same runs, the same compute, and the same clean-seed calibration set, and they are compared at a matched false-alarm rate. Two ablations isolate what is new. The first removes the off-diagonal term, keeping only convergent correlation, to test whether discriminant structure specifically matters. The second removes the shuffle interventions and keeps only the passive validity matrix. Beating the strongest baseline means a higher detection rate at 5% false alarms on the shortcut class of defects, meaning those the benchmark was designed to leave spatial plausibility intact, with detection on the other classes no worse.

Why not just…?

Why not just rank tools by accuracy on simulated pseudo-spots? This is the serious threat. Any defect that raises error on pooled imaging data will be caught this way, and our method has no claim on those defects. Two problems remain. First, pseudo-spots have no paired histology, and their neighborhood structure comes from binning. Defects that live in the image-to-spot mapping, or that let the image branch compensate for a dead gene branch, are therefore not exercised at all. Second, the Papi et al. result shows that some defects improve the default score. Our benchmark reports detection separately for each defect class, so if simulated accuracy matches our detector on the shortcut class, the experiment will show it and we will report that the psychometric machinery adds nothing there.

Why not just port DeepDiagnosis-style monitors? A detached gradient in one expert of a product-of-experts will register as unchanged weights, and we expect monitors to catch it. Gene misalignment, permuted reference labels, and barcode offsets produce perfectly healthy training curves, because the model is optimizing the wrong target correctly. Symptom monitors cannot see errors about what is being fit. The comparison of detection by defect class will measure exactly this.

Why not just run invariance tests, without a gold standard? Invariance tests catch defects that break a symmetry, such as dependence on batch size or tiling. They cannot catch defects that are perfectly consistent, such as a reference whose labels are permuted in the same way on every run. This is why expression shuffling compares against reference measurements rather than against the tool's own earlier output. It may turn out that invariance tests plus the expression-shuffle test catch nearly everything, and the passive validity matrix adds little. The second ablation is designed to find that out, and we would then recommend the cheaper combination.

Why not just use classical non-neural deconvolution and avoid the problem? Pipeline defects such as gene misalignment and double normalization affect CARD and regression methods just as much. Li et al. also report that probabilistic and deep tools lead on large datasets, and multimodal tools are the only way to use paired histology. CARD is included in the benchmark to measure how much of the defect exposure is specific to neural tools.

Proposed Experiments

The primary experiment runs every tool, defect, and seed combination on seqFISH+ and MERFISH pseudo-spots, where the validity matrix is computed against exact proportions. The next experiment uses the DLPFC Visium slices, where reference measurements come from layer annotations and held-out layer-marker genes excluded from deconvolution. Starfysh and GraphST are additionally run on the breast cancer cohort with paired H&E. For stereoscope and the mouse hippocampus and cerebellum, Allen Brain Atlas in situ images are registered to the tissue so that its qualitative validation becomes a quantitative validity matrix. The primary metric is detection rate per defect class at a 5% false-alarm rate, along with the area under the detection curve.

The first decisive check is whether the shortcut class of defects is caught by $D$ and expression shuffling significantly more often than by simulated RMSE. These defects include the barcode offset, the masked gene expert, and boundary bleed. The second check is whether the real defects recovered from issue histories are detected by our method at a rate similar to the injected ones. If both checks pass, we will run the detector on unmodified releases of all six tools and report measured prior reliance and discriminant scores. Any flagged release will be traced to its source before we make a claim, since a flag could also reflect a genuine modeling limitation rather than a bug.