Noise-Floor Relevance Margins for Agent-Run Single-Cell Differential Expression

Language-model agents now let biologists who do not program run a full single-cell RNA sequencing analysis by typing requests in plain English. Under the hood, these agents still call standard per-cell significance tests, and single-cell datasets contain tens or hundreds of thousands of cells. At that scale almost every gene reaches $p < 0.05$. The agent then writes a fluent report, and a non-expert reads "significant" as "important." We propose that before an agent makes any differential-expression claim, it measures how large an expression change the dataset can reliably tell apart from donor-to-donor and technical noise. It then refuses to call anything smaller "differential." Every gene lands in one of three plainly worded bins: meaningfully changed, practically unchanged, or not resolvable with these data.

In clinical measurement this is an old idea. A symptom score is only usable if the smallest change patients care about (the minimal clinically important difference, or MCID) is larger than the smallest change the instrument can distinguish from its own noise (the minimally detectable change, or MDC). Single-cell genomics has effect-size thresholds, but the threshold is either a convention such as $|\log_2 \mathrm{FC}| > 1$ or a number a programmer types in. Neither is tied to how noisy the specific dataset is. Our contribution is to estimate an MDC for each gene in each dataset, enforce "margin ≥ MDC" inside the agent's analysis step, and test whether this reduces false and non-replicating claims compared with simply swapping in a better statistical tool.

Core Claim

A relevance margin calibrated to a dataset's own donor-level noise floor, used by a Bayesian differential-expression test and reported in three bins, will reduce two things compared with both fixed-threshold and pseudobulk alternatives. The first is spurious "differential" calls on null comparisons. The second is non-replicating calls across independent studies of the same tissue. We also expect it to measurably reduce statistically fallacious statements in agent-written reports.

The part a reviewer should scrutinize is the calibration step, not the agent wrapper. If a sensible fixed threshold performs as well, the proposal reduces to a tool swap, and the experiments are designed to reveal that.

What Prior Work Does and Leaves Open

Several papers motivate the gap by documenting how significance testing misleads.

We reuse their three-way decision logic. All of these tools, however, take the margin as given. None of them handles count-valued, donor-structured biological data.

Two papers supply the statistical engine:

lvm-DE is the closest existing method and the one we build on. What it leaves open is where the threshold comes from, and whether a non-programmer ever sees the threshold at all.

Three agent papers define the accessibility layer and its weakness:

  • CellAgent splits the work among a planner, an executor that calls a curated toolkit, and an evaluator that inspects plots and clustering scores.
  • LAMBDA has language models query atlases, run statistical tests, and hold simulated expert debates.
  • scAgent targets cell annotation.

None of these checks whether a significant difference is large enough to matter. The dossier notes that their differential-expression steps default to standard Scanpy- and Seurat-style tests.

SC-Arena benchmarks single-cell reasoning, but it does not score statistical inference. We add that missing axis rather than compete on its existing tasks.

Two further papers constrain the design:

Objective study validity diagnostics provides the design pattern of a pre-specified diagnostic gate that must pass before any claim is released. We adopt that pattern for the MDC check.

Research Program

The method has three components.

1. Estimating the noise floor. For each gene $g$, we build null contrasts within a single condition. We split donors, not cells, into two random groups, repeatedly, and compute lvm-DE's posterior median log-fold change for each split. The per-gene MDC is the 95th percentile of the absolute null log-fold change across splits, written $\mathrm{MDC}_g$. When a condition has too few donors for donor splits, we fall back to cell-level splits within each donor. We flag that fallback in the report as a weaker guarantee.

2. Setting the margin. The margin used for gene $g$ is

$$\epsilon_g = \max(\mathrm{MCID}, \mathrm{MDC}_g),$$

where MCID is the smallest fold change the user says would matter. The agent elicits this in plain terms, for example "a change you would expect to confirm by qPCR or flow cytometry." If the user states nothing, it defaults to zero, so the margin falls back to the noise floor.

3. Three-way decision. Writing $\beta_g$ for the latent log-fold change, gene $g$ is:

  • meaningfully changed if the posterior probability $P(|\beta_g| > \epsilon_g \mid \text{data})$ exceeds a threshold chosen to control the false discovery rate, as in lvm-DE;
  • practically unchanged if $P(|\beta_g| < \epsilon_g \mid \text{data})$ exceeds the same threshold;
  • not resolvable otherwise, and this bin always includes genes where the user's MCID falls below $\mathrm{MDC}_g$.

Agent integration and report checking. We integrate this into an open reimplementation of a CellAgent-style planner, executor and evaluator, replacing the default differential-expression tool. The evaluator also checks every sentence of the draft report against the three bins. It rejects "trend toward significance" language, of the kind catalogued by Jin and colleagues ( A Retrospective Survey of Research Design and Statistical Analyses in Selected Chinese Medical Journals in 1998 and 2008). It rejects "no difference" claims for genes that are merely unresolvable. It rejects any "differential" claim for genes below the margin.

Scoring statistical claims. We will build and release a small benchmark of statistical-claim correctness for single-cell agents. It uses fixed analysis tasks and scores the reports automatically against the bins and a held-out replication set.

Baselines

Naive reference. The do-nothing reference is the analysis a current agent produces by default: a Wilcoxon rank-sum test per cell, Benjamini–Hochberg adjusted $p < 0.05$, reported as "differentially expressed." This is the behavior the dossier attributes to CellAgent's toolkit. We run it through the same agent shell on identical data, so that only the statistical step differs.

Conventional fixed cutoff. The same test plus the conventional filter $|\log_2 \mathrm{FC}| > 1$. This is what a careful non-expert following common tutorials would do.

Strongest per-cell method. lvm-DE with a fixed user threshold ( An empirical Bayes method for differential expression analysis of single cells with deep generative models). It is the strongest competitor because it already combines Bayesian effect-size testing with false discovery control on the same generative model we use. It differs from our method only in how the threshold is chosen. We run it with the same scVI model fits and the same number of training runs. We sweep its threshold over a grid (0.25, 0.5, 1, 2 in log-fold units). We then report both its best-in-hindsight setting and the setting a user would pick in advance, so it gets at least as much tuning as our method. The file gives no reported quantitative differential-expression result for lvm-DE; it claims superiority but reports no figure. No reported result found on Lacuna for lvm-DE on CELLxGENE-derived null or replication benchmarks.

Strongest classical competitor. Pseudobulk testing: summing counts per donor and applying a negative-binomial generalized linear model. It is the strongest non-Bayesian answer to donor-level pseudoreplication. We run it on the same donors and compare it at matched numbers of calls.

Ablations. Three ablations isolate the key ingredient:

  • replace $\mathrm{MDC}_g$ with a single dataset-wide median MDC, which tests whether gene-level calibration matters;
  • estimate MDC from cell splits instead of donor splits, which tests whether donor structure matters;
  • remove the evaluator's sentence-checking, which tests whether report-level checking adds anything beyond the statistical step.

What counts as winning. Beating a baseline means one of two things. Either lower false "meaningfully changed" calls on donor-split null contrasts at matched sensitivity. Or higher cross-study replication precision at a matched number of calls. In both cases the gap must hold across at least two tissues.

Why not just…?

Why not just give the agent lvm-DE with a default threshold? This is the real threat. It is one line of code and may capture most of the benefit. If a single fixed threshold matches our calibrated margin on null false calls and replication precision across tissues, then our calibration step is not needed. We would report the tool swap as the finding. The best-in-hindsight lvm-DE sweep is designed to settle this. We expect calibration to matter most where noise differs sharply between genes: lowly expressed genes and datasets with few donors. There, no single threshold is right for both noisy and quiet genes.

Why not just use pseudobulk testing? Pseudobulk fixes pseudoreplication. But it still returns $p$-values with no notion of practical equivalence, so it cannot tell "unchanged" from "underpowered." That distinction is the proof-of-the-null fallacy that Addressing Common Misuses warns about. It may match us on null false calls. The three-way experiment tests whether it can do the same on correctly labelling genes that are practically unchanged.

Why not just prompt the language model to avoid p-value fallacies? The survey in Not All Claims are Created Equal suggests that fluent confidence and correct understanding come apart even in experts. More concretely, a prompt cannot supply a noise floor the model has never computed. We include a prompt-only arm, meaning the default agent plus a statistics-aware system prompt. It is scored by the same automatic claim audit, so this question gets a direct answer.

Proposed Experiments

Data. We use CELLxGENE Census datasets, plus the Sikkema lung atlas and the Elmentaite intestinal atlas that LAMBDA used. We restrict to tissues and conditions with at least six donors per arm and at least two independent studies.

Experiment 1: null calibration. Contrasts built from donor splits within a single condition have no true differences. Any "meaningfully changed" call there is a false call. We report its rate against the number of cells, from 10k to 200k+. Our prediction is that the naive baseline's false-call rate grows with cell count while ours stays flat, because the margin rises with donor noise rather than shrinking with more cells.

Experiment 2: cross-study replication. We call genes in study A and measure what fraction replicate, meaning they land in the same bin, in study B of the same tissue and contrast. All methods are compared at matched call counts.

Experiment 3: report audit. We score agent reports from every arm on fixed prompts across the tissues. The metrics are the rate of sentences contradicting the bins, the rate of trend language, and the rate of null-as-proof claims. These are scored automatically, and a subset is scored by two blinded human raters.

Reference point from the literature. For the agent layer, the only benchmark number in the file concerns annotation, not inference: scAgent reports a Macro F1-score of 89.31% on a 1-million-cell, 35-tissue CELLxGENE subset (scAgent). We will check that swapping the differential-expression tool does not degrade SC-Arena task scores, using the SC-Arena codebase.

Decisive checks. The decisive check is Experiments 1 and 2 against the best-in-hindsight lvm-DE sweep. If calibration does not beat it in at least the low-expression and few-donor strata, the core claim fails.