Bulk RNA sequencing measures one averaged gene-expression vector from a tissue sample that contains millions of cells. Deconvolution methods try to recover which cell types are present and in what proportions, using a single-cell reference to define what each cell type looks like. These methods are known to fail in a systematic and silent way. In a benchmark of nine methods on 38,000 simulated breast-cancer mixtures, Tran et al. found that as the tumour fraction rises toward 95%, nearly every tool starts labelling normal epithelial cells as cancer epithelial cells, because the two lineages share most of their marker genes. They call this the "purity trap." The tools return a confident number either way, so a user cannot tell which samples have fallen into it.
This proposal argues that the purity trap is mainly a property of each individual sample: at high purity, that sample's counts barely constrain how the epithelial signal should be split. We propose to measure this directly. For each bulk sample, we fit a Laplace approximation (a Gaussian fitted to the curvature of the log-likelihood around its best fit) over that sample's cell-type proportions, under a count likelihood and a learned signature decoder. We then add the uncertainty in the signatures themselves. The output is not only an error bar. It is a posterior correlation structure that names which pairs of cell types the sample cannot tell apart. The obvious alternative is to put a Laplace approximation on the weights of a deep deconvolution network. We expect that version, along with ensembles and dropout, to miss the trap, because all of them measure uncertainty about the learned input-to-output mapping rather than about the information in the sample.
Core Claim
Deep deconvolution networks such as TAPE (Chen et al.) and Scaden map a bulk vector straight to a fraction vector. Methods that add uncertainty to such a network describe how much the trained mapping would change if training had gone differently. These include last-layer Laplace, used for domain adaptation in U-SFAN, as well as deep ensembles and Monte Carlo dropout. The purity trap is a different kind of failure: a bias shared by every reasonable mapping trained on the same simulated mixtures. Networks retrained from different random seeds should repeat the same mistake, so their disagreement should stay small exactly where the error is large.
Our claim is that the curvature of the sample's own count likelihood with respect to its proportions does carry the signal. When normal and cancer epithelium become hard to separate, the Hessian develops a nearly flat direction along their difference. A Laplace posterior then shows a strong negative correlation between those two fractions and a wide interval on the normal-epithelium fraction. This happens even though the point estimate is no different from a deterministic model's. The contribution is to make this computation work in the deep, adapted-decoder setting, at the throughput of a network-based method, and to show that it flags the trap while weight-space uncertainty does not.
Method
We keep a TAPE-style architecture. An encoder proposes proportions, and an interpretable decoder holds a gene-by-cell-type signature matrix that is adapted to the target cohort. We change two things.
First, we replace TAPE's continuous reconstruction loss with a negative-binomial likelihood on raw counts. This answers the concern raised by Count Bridges that treating counts as continuous values distorts sparse data.
Second, we treat the encoder output only as a starting point. For each sample we refine it by maximizing the likelihood over proportions. We write the proportions as $\pi = \mathrm{softmax}(z)$ with $z \in \mathbb{R}^{K-1}$ in additive log-ratio coordinates, so that a Gaussian over $z$ never puts mass outside the simplex. This plays the role that the Softplus transform plays in MEAD.
The per-sample posterior covariance combines two sources of uncertainty:
$$ \Sigma_z \approx H_z^{-1} + J\,\Sigma_\phi\,J^\top . $$
The first term, $H_z$, is the negative Hessian of the sample's negative-binomial log-likelihood plus a weak prior, taken with respect to $z$ at the refined optimum. It measures how strongly this sample's counts pin down its proportions. The second term carries uncertainty from the decoder signatures. Here $\Sigma_\phi$ is a Kronecker-factored Laplace covariance over the decoder signature weights $\phi$, following Ritter et al., fitted once on the reference pseudo-bulks and the adapted cohort. $J = \partial z^\ast / \partial \phi$ is the sensitivity of the optimum to the signatures, which the implicit function theorem gives as $-H_z^{-1}\,\partial^2 \ell / \partial z\,\partial\phi$, where $\ell$ is the log-likelihood. With $K$ cell types, the per-sample cost is one $(K-1)\times(K-1)$ solve. That keeps the method near TAPE's reported speed of about 800 samples in 120 seconds.
From $\Sigma_z$ we report two outputs. The first is an interval for each fraction, obtained by pushing samples of $z$ through the softmax. The second is a confounding score for each pair of cell types: the posterior correlation of the two fractions, together with the share of the normal-epithelium variance that lies along the most uncertain eigenvector of $\Sigma_z$.
We take seriously the known failure modes of Laplace approximations in deep models (Unveiling Empirical Pathologies) and under misspecification (Nonasymptotic Laplace under misspecification). Our Gaussian lives over only $K-1$ variables rather than millions of weights, which should make the approximation more trustworthy. Still, near-zero fractions and flat directions are exactly where a Gaussian can be wrong. We therefore check it directly against Hamiltonian Monte Carlo on the same low-dimensional likelihood, as described in the experiments below.
Research Program
The work has three parts.
The first part builds the count-likelihood deconvolver and the per-sample Laplace step described above, using marginal-likelihood tuning of the prior precision. Tooling exists for this, for example robust-sbl.
The second part tests the purity trap directly. We regenerate the Tran et al. design from the Wu breast-cancer atlas (100,064 cells, 26 patients, ER+/HER2+/TNBC), with purity running from 5% to 95%. We train on mixtures built from one set of patients and test on mixtures built from held-out patients. We then ask two questions: does the confounding score rise with purity, and does it predict the size of the normal-epithelium error?
The third part tests shift. We take the model trained on Wu and apply it to pseudo-bulks built from the independent Bassez and Pal cohorts, and to a held-out-cell-type setting in which one immune population is removed from the reference. We also check the islet setting, using cross-platform references (Xin and Segerstolpe) against the 77-donor bulk cohort.
Baselines
The naive reference point is Scaden and TAPE run as published, giving point estimates only, plus a residual-based flag. The flag is the bulk reconstruction error of the fitted model, thresholded on held-out pseudo-bulks. It costs nothing, so any value our method adds must exceed it.
The strongest existing competitors on the interval question are BayesPrism and MEAD.
BayesPrism is a Gibbs-sampling Bayesian deconvolver included in the Tran benchmark. It was among the methods that limited false positives best, yet it still degraded at high purity. It is the most direct threat, because it already produces a posterior.
MEAD gives sandwich-based confidence intervals in a GLM that adjusts for platform scaling. It is the strongest frequentist answer to the same question.
All methods receive the same reference cells, the same marker-gene set, the same pseudo-bulks, and a matched hyperparameter tuning budget on a shared validation split. We report wall-clock time on the same hardware. The file contains no reported per-purity error or coverage numbers for any method on the Wu pseudo-bulks: no reported result found on Lacuna for BayesPrism, MEAD, or Scaden on this benchmark. We therefore rerun all of them ourselves. The only externally reported accuracy anchor is SCDC's Pearson correlation of 0.991 with ground truth on the 6:3:1 MDA-MB-468/MCF-7/fibroblast cell-line mixture (SCDC). We use it as a sanity check that our point estimates are competitive, not as the target.
Our method counts as beating BayesPrism if it matches BayesPrism's interval coverage for normal epithelium at purities of 80% and above while being much faster. It counts as beating MEAD if it gives better coverage under cross-cohort shift, where MEAD's linear signature model cannot adapt.
Weight-space uncertainty baselines are run on the same backbone. These are a five-member deep ensemble, Monte Carlo dropout, and a last-layer Laplace over the encoder in the style of U-SFAN. Together they make up the obvious "Laplace plus deconvolution" combination.
Two ablations isolate the key ingredient. The first removes the signature term $J\Sigma_\phi J^\top$, which tests whether reference uncertainty matters under shift. The second replaces the full $\Sigma_z$ with its diagonal, which removes the confounding score and tests whether the correlation structure, rather than the interval width, is what flags the trap.
Why not just…?
Why not just use a deep ensemble or Monte Carlo dropout on Scaden? These are cheap, and in vision they are often competitive. Our argument is that the purity trap is a bias shared by every model trained on the same mixtures, so members should agree with each other while all being wrong. This is a testable prediction. If ensemble spread predicts normal-epithelium error at high purity as well as our confounding score does, measured by the area under the ROC curve for flagging samples whose error exceeds 10 percentage points, then the per-sample likelihood view adds nothing and we will report that. Ritter et al. found that dropout stayed overconfident on unfamiliar inputs, but that result comes from images and does not settle this case.
Why not just use BayesPrism, which is already Bayesian? This is a real threat. If BayesPrism's posterior already widens in the trap regime, our contribution narrows to speed and to working with an adapted deep decoder. The Tran benchmark shows that BayesPrism's point estimates still fail at high purity. It does not show whether BayesPrism's intervals cover the truth there. Our coverage-versus-purity curve answers that. We will report honestly if BayesPrism is already calibrated.
Why not just use MEAD's sandwich intervals? MEAD gives valid intervals when its linear model and fixed signatures hold. The threat it faces is cross-cohort shift, where the signatures themselves are wrong. The Bassez/Pal transfer experiment and the removed-cell-type experiment compare coverage directly. If MEAD matches us there, a deep decoder is unnecessary for this question.
Why not just threshold the reconstruction residual? A missing cell type should raise the residual. Mis-splitting two lineages that share signatures need not, because the bulk can be reconstructed equally well either way. We expect the residual to catch the removed-cell-type case and miss the purity trap, and we expect the confounding score to show the reverse pattern. Comparing the two methods' ability to flag samples in each regime tests this.
Decisive Checks
The central experiment plots, against purity, three quantities for the normal-epithelium fraction on held-out-patient Wu mixtures. The first is the empirical coverage of the nominal 90% interval. The second is the AUROC for flagging large errors. The third is selective error: the mean absolute error after the 20% least certain samples are set aside. Each is shown for our method, both ablations, the weight-space baselines, BayesPrism, and MEAD. The proposal succeeds if three things hold together: the full-covariance Laplace keeps coverage near 90% above 80% purity, encoder-weight uncertainty does not, and the diagonal ablation loses most of the flagging power.
A fidelity check compares $\Sigma_z$ with Hamiltonian Monte Carlo on the same likelihood for 500 samples. We measure covariance error and coverage agreement, with particular attention to fractions below 2%, where a Gaussian over log-ratios may break down. The same check run under the count likelihood answers whether a continuous Gaussian is adequate for sparse counts.
A downstream check carries the intervals into two cohort analyses. In the 77-donor islet cohort, we test whether the HbA1c–beta-cell association reported by SCDC at p = 0.0019 survives once deconvolution uncertainty is included. In TCGA breast cancer (1,038 samples), we test whether associations involving normal epithelium change once high-confounding samples are down-weighted. This shows whether the flags change scientific conclusions or only the error bars.
References
- Performance of tumour microenvironment deconvolution methods in breast cancer using single-cell simulated bulk mixtures
- Deep autoencoder for interpretable tissue-adaptive deconvolution and cell-type-specific gene analysis
- Uncertainty-Guided Source-Free Domain Adaptation
- Count Bridges enable Modeling and Deconvolving Transcriptomic Data
- Statistical inference for cell type deconvolution
- A Scalable Laplace Approximation for Neural Networks
- Identity Curvature Laplace Approximation for Improved Out-of-Distribution Detection
- Nonasymptotic Laplace approximation under model misspecification
- SCDC: Bulk Gene Expression Deconvolution by Multiple Single-Cell RNA Sequencing References