A bulk RNA measurement averages over thousands of cells. "Deconvolution" methods try to recover what fraction of a tissue sample each cell type makes up. They do this by comparing the bulk profile against reference expression profiles, called signatures, estimated from single-cell data. Every such analysis quietly makes a modeling simplification first: it decides which cell types count as distinct. Some subtypes get lumped into one category and others are kept apart. Today that choice is made by hand or copied from an annotation tree. Yet the right choice depends on two things. One is how distinguishable the subtypes are. The other is how sparse the single-cell reference is, because most entries are zeros caused by capture failure rather than real silence. A rare stem-like subpopulation captured in 40 shallow cells has a signature so noisy that "estimating its proportion" can mean reporting noise as biology.
We propose to make the lumping decision a statistical output rather than an input. For a given bulk cohort and reference atlas, the method returns the finest partition of cell states whose proportions can actually be estimated, with error bars on each. It also gives an explicit statement of which distinctions are masked, and why. The masking can come from the bulk measurement or from the sparsity of the reference.
Core Claim
Our central claim is that uncertainty in the reference signatures, driven by dropout and small cell counts, often matters more than bulk measurement noise when the populations are rare and fine-grained. Current deconvolution treats the signatures as fixed. If the reference uncertainty is propagated with a count-based model, and used to merge cell types along the Cell Ontology tree until each remaining proportion is estimable, the method should do two things that fixed-resolution deconvolution cannot. It should give correctly calibrated error bars at whatever resolution it reports. It should also predict, before any bulk data are analyzed, the minimum detectable proportion of a rare population as a function of reference depth, reference cell number, and how distinct the signature is.
This turns a familiar lesson from systems biology into a concrete procedure. Witt et al. showed that mechanistic models which keep every process lose parameter identifiability. Lumping, as in Stumpf's merging of PspB and PspC into one complex, restores identifiability at the cost of resolution. Cell-type lumping in deconvolution is the same trade-off, and nobody currently optimizes it.
Relation to Prior Work
Several papers motivate the gap. Palmer et al. and Lin et al. show that the clinically relevant signal can sit in rare stem-like cells that bulk averages hide. Du et al. note that tissue models routinely omit rare populations altogether. Yuan et al. frame the core ambiguity of sparse single-cell data: a zero may be true silence or a capture failure. A constraining result comes from Symphony, A Nonparametric Multi-view Model for Estimating Cell Type-Specific Gene Regulatory Networks. It reports that its deconvolution collapses for populations below about 5%. That is evidence a hard floor exists, but nobody has characterized that floor as a function of reference quality.
The method draws on three sources. MEAD, Statistical inference for cell type deconvolution, supplies identifiability conditions under platform scaling differences, noise-weighted genes, and sandwich-type error bars. However, it works at a fixed, predefined partition of cell types. SCDC, SCDC: Bulk Gene Expression Deconvolution by Multiple Single-Cell RNA Sequencing References, deconvolves hierarchically along a tree to separate similar subtypes. Its tree and stopping depth are fixed in advance, and it treats references as continuous matrices. The negative binomial treatment of overdispersed counts in scNBMF, A fast and efficient count-based matrix factorization method, supplies our model for signature uncertainty. POPPY, Cross-modal transfer learning for mapping bulk transcriptomes at cellular level, shows that the Cell Ontology tree is a useful distance structure for bulk-to-cell mapping. We reuse that tree as the space of allowed merges, not as a loss.
Generative unmixers such as Count Bridges, Count Bridges enable Modeling and Deconvolving Transcriptomic Data, and TAPE, Deep autoencoder for interpretable tissue-adaptive deconvolution, aim at richer outputs: single-cell counts or sample-specific signatures. They do not ask whether a fine-grained distinction is supported at all. TAPE's own authors report difficulty with rare and highly similar subtypes.
We deliberately do not pursue the more obvious bridge between these directions, which would place attractor-landscape or neural ODE dynamics inside a count deconvolver. The Attractome work, Attractome, is explicitly untested on real cohorts. Building on it would stack an unvalidated abstraction on an unresolved identifiability problem.
Research Program
The method has three parts.
The first part estimates signature uncertainty from counts. For each leaf cell state in the reference, we fit a negative binomial model per gene, accounting for each cell's library size. This yields a posterior over that state's mean expression, so a state with few, shallow cells gets a wide posterior. We check this estimate against a cell-level bootstrap of the reference.
The second part propagates both sources of error into the proportion estimates. For a candidate partition $P$, let $\hat p_P$ be the weighted least-squares or MEAD-style proportion estimate. Let $S_P$ be its genes-by-types signature matrix and $W$ the gene weights. To first order (the delta method), the error variance splits into a bulk term and a reference term:
$$ \mathrm{Var}(\hat p_P) \approx \underbrace{(S_P^\top W S_P)^{-1} S_P^\top W \Sigma_y W S_P (S_P^\top W S_P)^{-1}}_{\text{bulk noise}} \;+\; \underbrace{J_P\, \Sigma_{S_P}\, J_P^\top}_{\text{reference sparsity}} $$
Here $\Sigma_y$ is the bulk residual covariance, $\Sigma_{S_P}$ is the posterior covariance of the signature entries from the first part, and $J_P$ is the derivative of $\hat p_P$ with respect to the signature entries.
The third part chooses the partition. We start at the leaves of the Cell Ontology tree restricted to states present in the reference, then greedily merge sibling pairs. Each step merges the pair whose split is least supported, measured by the ratio of the estimated proportion to its standard error, averaged over the cohort. We stop when every remaining type passes a pre-registered estimability threshold. A by-product is a predicted detection limit for each leaf. Solving the variance expression for the proportion at which the signal-to-error ratio reaches the threshold gives a closed-form curve in reference cell count, depth, and signature contrast. This is a falsifiable prediction we test directly.
Baselines
The naive reference point is deconvolution at a single fixed resolution. One version runs at the finest annotation available. The other runs at a coarse major-lineage level chosen by an analyst. Both use the same reference, genes, and solver as our method, so that only the partition differs. Our method beats this if it matches the coarse level's accuracy on the types it merges while keeping more fine distinctions at equal error. It also beats this if it avoids the finest level's false rare-type detections.
The strongest existing competitor for proportion accuracy is SCDC, because it already handles multiple references and hierarchical subtypes. Its best reported number in the file is a Pearson correlation of $r = 0.991$ with ground-truth proportions on the 6:3:1 MDA-MB-468/MCF-7/fibroblast RNA mixture, from SCDC. That benchmark has only three well-separated types, so we will match rather than beat it there. SCDC is also the only method in its comparison to recover the negative correlation between HbA1c and beta-cell proportion in the 77-donor islet cohort ($p = 0.0019$). We expect to reproduce that association while additionally reporting which islet subtypes are estimable. SCDC runs with its own tree and with the same pooled references we use.
The strongest competitor for calibrated error bars is MEAD. The file gives no reported numerical result on Lacuna for MEAD's accuracy, only that it is competitive with CIBERSORT, MuSiC, and OLS on islets. We compare against it at the finest partition and at our selected partition.
The key ablation removes the reference-sparsity term from the variance expression, keeping everything else identical. If merging decisions and coverage barely change, our central claim is wrong. In that case bulk noise dominates and existing error bars suffice.
All methods share data splits, gene sets, and an equal hyperparameter search budget. Our merge threshold is fixed on synthetic data before any real cohort is touched.
Why not just…?
Why not just run MEAD at the finest resolution and merge cell types whose confidence intervals are wide? This is the most serious alternative, and it may be almost as good. It differs from our method only in ignoring reference uncertainty. If MEAD's error bars already absorb most of the error, this simple rule will select the same partitions. The ablation above is exactly this comparison. The decisive measurement is error-bar coverage for rare spiked-in types as the reference is downsampled. If MEAD-based merging keeps nominal coverage at 30 reference cells per type, we will report that the cheaper rule suffices and recast the contribution as the detection-limit theory alone.
Why not just impute the reference first with SERM or a denoiser, then deconvolve finely? Imputers such as SERM and the embedding denoiser DICE, Clustering by Denoising, borrow strength across cells. That borrowing can make rare-state signatures look sharper than the data support. The field already recognizes this risk of over-smoothing and invented correlations. We test it directly. We impute downsampled references, deconvolve at full resolution, and measure false detection of rare types absent from the bulk. If imputation improves accuracy without inflating false positives, we fold it into our first part as a better signature estimator rather than treating it as a rival.
Why not just use a generative unmixer such as Count Bridges or TAPE, which outputs cell-level profiles and sidesteps lumping? Producing more detailed output does not create information the bulk and reference lack. A rare type below the detection floor will be filled in by the generative prior. We run both methods on the same spike-in series and score their rare-type proportions against truth. We expect them to fall below our predicted floor wherever they report fine distinctions. If they beat our predicted floor, our theory is wrong.
Proposed Experiments
The main testbed is semi-synthetic bulk data with known composition. We build pseudo-bulks from the Curated Cancer Cell Atlas (3CA; 2.7 million cells, 15 cancer types) and from PBMC data. Holding out cells for the references ensures that reference cells never enter the bulk. We plant a chosen subtype at 0.5, 1, 2, and 5%. In a separate step, we downsample the reference subtype to 20–500 cells and lower its read depth, which varies sparsity on its own. The key quantities are four. The first is the error in estimated proportion for each type in the selected partition. The second is the coverage of 90% error bars. The third is the rate of false rare-type detection when the subtype is absent from the bulk. The fourth is the agreement between the predicted and empirical detection limits across the grid.
The decisive check is that last one. The closed-form detection-limit curve should match the empirically measured floor within a factor of two across reference depths. In particular, it should explain Symphony's reported collapse below 5% as a consequence of reference sparsity rather than a fixed constant.
We next test on the Kassandra simulated bulks (2,286 profiles) for proportion accuracy under realistic sorted-cell mixtures. We also use the SCDC in vitro mixture as a sanity check where all methods should be near ceiling.
The real-cohort test uses human pancreatic islets. The references are Xin and Segerstolpe, and the bulk is the 77-donor cohort. We check that the selected partition recovers the HbA1c–beta-cell association. We also check whether that partition stays stable when the two references are swapped. A stable partition is evidence that the method reflects biology rather than one lab's sparsity. As an exploratory step, we apply the selected partition to the 330 melanoma immunotherapy bulks. There we ask whether the T-cell and NK-cell subsets that POPPY flags as response drivers are estimable as separate types at all, or whether they must be lumped.
References
- Mechanism of PP2A-mediated IKKβ dephosphorylation: a systems biological approach
- From qualitative data to quantitative models: analysis of the phage shock protein stress response in Escherichia coli
- A gene expression profile of stem cell pluripotentiality and differentiation is conserved across diverse solid and hematopoietic cancers
- Integrative analyses of gene expression and DNA methylation profiles in breast cancer cell line models of tamoxifen-resistance indicate a potential role of cells with stem-like properties
- The Interplay between Wnt Mediated Expansion and Negative Regulation of Growth Promotes Robust Intestinal Crypt Structure and Homeostasis
- Challenges and emerging directions in single-cell analysis
- A Nonparametric Multi-view Model for Estimating Cell Type-Specific Gene Regulatory Networks
- Statistical inference for cell type deconvolution
- SCDC: Bulk Gene Expression Deconvolution by Multiple Single-Cell RNA Sequencing References
- A fast and efficient count-based matrix factorization method for detecting cell types from single-cell RNAseq data
- Cross-modal transfer learning for mapping bulk transcriptomes at cellular level
- Count Bridges enable Modeling and Deconvolving Transcriptomic Data
- Deep autoencoder for interpretable tissue-adaptive deconvolution and cell-type-specific gene analysis
- Attractome: From Theory to Generative Models for Cancer Dynamics and Control
- Leveraging data-driven self-consistency for high-fidelity gene expression recovery
- Clustering by Denoising: Latent plug-and-play diffusion for single-cell embeddings