Limits of Detection for Rare Immune Cells in Bulk RNA-seq

When a tumor biopsy is ground up and sequenced as one sample ("bulk" RNA sequencing), computational deconvolution tries to recover how much of the tissue was made of each cell type. These methods estimate the mix of cancer cells, stroma and immune subsets from the pooled expression profile. Clinical studies now treat those estimates as measurements. A deconvolution output of "0% regulatory T cells" or "2% NK cells" is correlated with survival or immunotherapy response across thousands of archived samples. Such a number mixes up two very different states. The cells may really be absent. Or they may be present, but the sample was sequenced too shallowly, and was too dominated by tumor cells, for their marker transcripts to be counted at all. No current method reports which of the two applies.

We propose to make that distinction explicit. The central object is a sample-specific limit of detection for each cell type: the smallest fraction that could be reliably distinguished from zero. It depends on the sample's read count, its tumor purity, and how different the cell type's expression is from the tumor background, and all three are computable before or after sequencing. We will derive this limit from a count-level likelihood and pair it with a boundary-aware test that labels each estimate as detected, not detected, or below the limit of detection. Then we check whether the predicted limit matches the empirical failure boundary of existing deconvolution methods on benchmarks where depth is lowered by read thinning. If it does, the limit becomes both a reporting standard for retrospective cohorts and a design rule for how deep to sequence.

Core Claim

Detectability of a rare immune fraction in bulk data is governed mainly by one quantity, which we will call the detection information. It is roughly the number of reads times the signature contrast between that cell type and the tumor background, after accounting for other, similar cell types. Existing methods neither compute nor report it.

Two consequences follow. First, "zero" calls from current methods systematically conflate absence with undetectability in high-purity or shallow samples. Second, because counts are overdispersed (noisier than a pure counting process), sequencing deeper stops helping at a depth we can predict. This is a testable claim, and it matters for trial design.

Why This Gap Is Open

Two literatures meet here without talking to each other.

The low-abundance transcript literature establishes that finite depth, not biology, often decides what is observed. Lonsdale et al.'s GTEx paper documents that standard depths fail to reproducibly quantify low-concentration transcripts. Mock et al. show that shallow nanopore budgets cause complete dropout of low-abundance transcripts in clinical samples. Moldovan et al. show that signals from rare blood populations are swallowed by dominant cells. These papers motivate the proposal: they establish the mechanism but never connect it to cell-type fractions.

The deconvolution literature, in contrast, evaluates methods almost entirely on mixtures in which depth is not a controlled variable. The closest work, the breast cancer tumor-microenvironment benchmark, makes 38,000 simulated mixtures from the Wu et al. atlas across 5–95% purity. It finds that BayesPrism and DWLS have the fewest combined false positives and negatives on granular immune lineages, and it names a "purity trap" at high purity. However, its mixtures are linear sums of single-cell counts, and it oversamples rare cells with SMOTE (a synthetic oversampling technique). It therefore never asks what happens when a rare cell's transcripts are physically missing from the library. That paper both motivates our work (the purity trap) and constrains it: we inherit its purity grid and its winners as baselines.

Three methods already attend to noise near zero, and each stops short of a detection limit.

  • MEAD gives valid confidence intervals for proportions near the boundary and weights genes by inverse noise variance. It assumes that gene-specific platform scaling is unrelated to expression level, and its authors flag that this may fail under dropout, which is exactly our regime.
  • Count Bridges models integer counts directly and outperforms CIBERSORTx and MuSiC on bulk proportions. It was not tested under extreme purity or controlled depth.
  • SCDC ensembles references across platforms. It assumes reconstruction error tracks proportion accuracy.

None of the three produces a per-sample, per-cell-type statement of what could have been seen. Our method reuses their count likelihoods and inverse-variance gene weighting.

Research Program

We write the bulk read count for gene $g$ in a sample of total depth $D$ as negative-binomial, with mean $D\,m_g(w)$, where $m_g(w)=\sum_c w_c x_{gc}$. Here $x_{gc}$ is the normalized reference profile of cell type $c$, $w_c$ is its fraction, and $\phi_g$ is the gene's overdispersion.

Moving a small fraction $\varepsilon$ of the sample from tumor ($T$) to cell type $c$ yields detection information

$$I_c(D)=D\sum_g \frac{(x_{gc}-x_{gT})^2}{m_g\,(1+\phi_g D m_g)} ,$$

which is then reduced by its Schur complement against the other cell types. That reduction is how collinear lineages such as CD4 T cells, CD8 T cells, regulatory T cells and NK cells cost information.

The limit of detection at false-positive rate $\alpha$ and power $1-\beta$ is then

$$\mathrm{LoD}_c \approx \frac{z_{\alpha}+z_{\beta}}{\sqrt{I_c(D)}} .$$

This follows from the standard large-sample normal approximation to the maximum-likelihood estimate. The formula predicts two regimes. The limit falls as $D^{-1/2}$ when counting noise dominates. It plateaus once $\phi_g D m_g \gg 1$ for the informative genes, because extra-counting noise then dominates. Purity enters through $m_g$, which is inflated by tumor-dominant genes.

Beyond the formula, we will build four things:

  1. A calibrated estimator of $\phi_g$ from bulk cohorts. We will estimate overdispersion from the bulk cohorts themselves rather than from single-cell references. This sidesteps the reference–bulk scaling mismatch that MEAD flags.
  2. A boundary-aware likelihood-ratio test for $w_c=0$. The test uses the mixture-of-chi-square null appropriate when a parameter sits on the boundary of its allowed range. It outputs a three-way call: detected, absent, or below the limit of detection.
  3. A method-agnostic wrapper. It attaches the limit to the point estimates of any deconvolution tool.
  4. A depth-controlled benchmark. Real bulk libraries and pseudo-bulks are binomially thinned to target read counts, so depth is varied without changing composition.

The key design choice is to thin reads rather than rescale normalized expression. That is what makes low-abundance markers actually vanish.

Baselines

Naive reference. The naive reference is the existing practice: run a deconvolution tool on the full-depth or thinned data and treat any estimate below a fixed threshold (for example 1%) as absent, with no depth or purity dependence. Beating it means our three-way calls achieve lower false-absence rates at matched false-presence rates across the depth-by-purity grid.

BayesPrism. The strongest existing method for this question is BayesPrism, a Bayesian Dirichlet-multinomial model that uses single-cell references and yields posterior uncertainty on counts. The breast benchmark reports that it and DWLS have the lowest combined false positives and negatives on granular immune lineages, and that it is among the most purity-stable methods. That paper summary gives no numeric error rates; no reported result found on Lacuna for BayesPrism's rare-lineage error rate. We run it on identical thinned mixtures with the same Wu et al. reference. We treat "posterior 95% interval excludes zero" as its detection call and give it the same reference preprocessing budget as our method. Beating it means better-calibrated false-absence rates below roughly 5% true fraction, and a limit of detection that predicts where its calls fail.

MEAD. We compare against MEAD's confidence intervals as the inference-focused competitor, run with its default inverse-variance weighting. It is the right baseline because it already targets valid inference near zero proportions. We expect its interval coverage to degrade specifically at shallow depth, where dropout makes scaling depend on expression level. That degradation is the prediction to confirm or refute.

Count Bridges. We include Count Bridges as the count-native generative competitor.

Controlled physical ground truth. SCDC reports Pearson 0.991 against the 6:3:1 cell-line mixture (in SCDC). We use that mixture, thinned, to check that our wrapper does not degrade point accuracy at full depth.

Ablations. Two ablations isolate the key ingredient. The first sets $\phi_g=0$, a pure Poisson model, which should wrongly predict that the limit keeps falling with depth. The second computes information without the Schur-complement collinearity correction, which should overstate detectability of T-cell subsets.

Why not just…?

Why not just use BayesPrism's posterior intervals? This is the main threat. If BayesPrism's intervals are already well calibrated near zero at shallow depth and high purity, then our test adds little as an estimator. Our experiments measure its false-absence rate as a function of true fraction and depth. What its posteriors do not provide is a pre-sequencing prediction: a closed form telling a trial designer what depth is needed to detect 2% regulatory T cells at 80% purity. If BayesPrism's calls prove calibrated, we will reframe the contribution around the limit formula and the benchmark rather than the test.

Why not just use MEAD's confidence intervals? MEAD already handles the boundary, but it relies on scaling being unrelated to expression. Its own authors say this may break under dropout. The thinning grid tests this directly by measuring interval coverage at each depth. Should MEAD's coverage hold, we adopt its intervals inside our wrapper.

Why not just sequence deeper? Deeper sequencing is often impossible for archived TCGA or GEO cohorts. The overdispersion term also predicts that it stops helping. The experiment that settles this compares the empirical depth at which each method's false-absence rate stops improving with the predicted plateau depth. If no plateau appears, the practical advice becomes simply "sequence deeper," and we will say so.

Why not just use a targeted spatial panel? SpatialCTD shows that spatial imaging panels of about 1,000 genes give exact ground truth. Such panels do not exist for retrospective bulk cohorts, which are the cohorts where the conflation of absence and undetectability corrupts clinical associations.

Proposed Experiments

Main test. We generate pseudo-bulks from the Wu et al. breast atlas across the benchmark's 5–95% purity grid, replacing summation of normalized profiles with summation of raw counts followed by binomial thinning to depths from 1 to 50 million reads. For each method, we fit the empirical boundary where false-absence exceeds 20% and compare it with the predicted limit of detection per cell type. The decisive check is whether the predicted limit ranks cell types and conditions correctly: a rank correlation across method, cell type, purity and depth. It must also capture the predicted plateau. A limit that tracks one method but not others would mean we have described a method artifact rather than an information limit.

Physical mixtures. Thinning the SCDC cell-line mixture tests the same prediction without single-cell simulation artifacts.

Real bulk. We thin the 1,038 TCGA breast profiles, using full-depth three-way calls as reference. We ask what fraction of immune "zeros" in shallow libraries sit below the limit of detection. We then check whether excluding below-limit samples changes published-style associations between immune fractions and subtype.

Spatial ground truth. SpatialCTD pseudo-spots supply exact truth in immune-rich lung, kidney and liver tissue, under the caveat that its targeted panel removes the lowest-abundance genes.

Failure mode. A clear negative outcome would be predicted limits that miss empirical boundaries by more than a factor of two. That would suggest that reference–bulk platform mismatch, not counting information, dominates detection.