When Can a Microbiome Survey See a Single Dysbiosis Axis?

Many microbiome studies hope to find one dominant direction of change in the community: a coordinated shift in which a group of bacteria rises while others fall across a subset of hosts or sites. These shifts are usually called dysbiosis or an ecological gradient. The data come from 16S rRNA gene sequencing, which counts barcode-like gene fragments per taxon per sample. The resulting matrix is mostly zeros, its rows have very different sequencing depths, and it only reports proportions, not absolute amounts. Analysts run log-ratio PCA or distance-based ordination on it and read off the first axis. Nobody can say beforehand whether a given cohort is large enough, or deep enough, for that axis to be real rather than noise. Random matrix theory has a precise answer to this question for idealized data. Below a sharp signal-strength threshold, the top principal component of a "signal plus noise" matrix carries no information about the signal at all.

We propose to bring that answer to amplicon count data. The key observation is that in count data the noise level of each entry is not an unknown nuisance. Under Poisson or negative-binomial sampling, the variance of each entry is set by its expected count. To first order that expected count is a rank-one outer product: sample depth times baseline taxon abundance. Both factors can be estimated from the row and column sums. The variance profile, which existing optimal spectral methods must assume is known, can therefore be plugged in from the marginals. We will turn this into two deliverables. The first is a "recovery calculator" that predicts, from library sizes and taxon prevalences alone, the minimum effect strength a single dysbiosis axis needs before any spectral method can find it. The second is an estimator that reweights the count matrix by its plug-in variance profile and recovers the axis down to that predicted limit.

Core Claim

Write the samples-by-taxa count matrix as $Y \in \mathbb{N}^{n \times p}$, with $N_i$ the library size of sample $i$ and $q_j$ the baseline relative abundance of taxon $j$. A single dysbiosis axis is a multiplicative perturbation:

$$\mathbb{E}[Y_{ij}] = N_i q_j \exp(\theta\, u_i v_j), \qquad \mathrm{Var}(Y_{ij}) = \mu_{ij}(1 + \phi_j \mu_{ij}).$$

Here $u$ holds the unit-norm sample scores (which hosts are dysbiotic, and how much), $v$ holds the unit-norm taxon loadings (which taxa move), $\theta$ is the effect strength, $\mu_{ij}$ is the expected count, and $\phi_j$ is a taxon-specific overdispersion. Take the centered matrix $R_{ij} = Y_{ij} - N_i \hat q_j$ and expand for small $\theta$. The result is a rank-one spike $\theta\, N_i q_j u_i v_j$ observed through noise whose entrywise variance profile $S_{ij} = N_i q_j(1+\phi_j N_i q_j)$ can be estimated from marginals and a per-taxon dispersion fit.

This puts amplicon data inside the variance-profile spiked models of Spectral Phase Transition and Optimal PCA in Block-Structured Spiked Models. That paper shows that reweighting entries by the inverse noise variance makes an informative outlier eigenvalue appear exactly when the top eigenvalue of a small $K \times K$ matrix built from the variance profile exceeds 1. We will bin samples by depth quantile and taxa by abundance quantile. This builds the analogous $K \times K$ matrix directly from $\hat N$, $\hat q$, $\hat\phi$, giving a predicted threshold $\theta^\star$ for each dataset. The claim to test has two parts. The first is that $\theta^\star$ predicts where recovery actually begins on realistic 16S data. The second is that inverse-variance reweighting recovers the axis well below where log-ratio PCA and square-root-variance (Pearson residual) PCA do. What is new is a computable detection limit for amplicon surveys, together with a threshold-derived rule for the prevalence and depth filtering that practitioners currently set by habit.

Research Program

The theory step adapts existing results rather than proving new ones. The block-structured result of Mergny, Ko and Krzakala is stated for symmetric Wigner-type matrices. Our matrices are rectangular, and the profile is continuous before binning. We will state the rectangular block extension as a conjecture and verify it numerically, in the same way that Gradient Descent Dynamics of Rank-One Matrix Denoising confirms its rectangular threshold $\lambda = c^{1/4}$ on $900 \times 1200$ Gaussian matrices. We then check how the predicted $\theta^\star$ depends on $K$, so the answer is not an artifact of the binning.

Two features of 16S data push beyond those models, and each is a separate thing we will build. The first is sparsity. Rare taxa sampled at low depth produce entries that are almost always zero and occasionally large. PCA recovery thresholds in low-rank matrix inference with sparse noise shows that sparse noise creates its own outlier eigenvalue. The signal must first leave the bulk and then overtake this noise outlier, a two-step transition with a closed-form critical value in random regular graphs. We expect inverse-variance weighting to amplify exactly these low-$\mu_{ij}$ entries. That would create localized spurious components concentrated on a handful of rare taxa or shallow samples. So we will cap the weights. We will also choose a prevalence and depth cutoff that minimizes the predicted $\theta^\star$, trading the information lost by dropping taxa against the noise outlier they create. That cutoff becomes the filtering rule.

The second feature is compositionality. Because counts are tied to $N_i$, a rise in one dominant taxon depresses all others. In our model this appears as a rank-one correction along $q$ that is absorbed by centering on $N_i \hat q_j$. We will check that the residual spike is still recovered in log-ratio terms, reporting loadings after a centered log-ratio map. The point is to make the output interpretable in the Aitchison geometry argued for in Aitchison Embeddings for Learning Compositional Graph Representations, which embeds graph nodes on the simplex via isometric log-ratios. We use only its geometric argument, not its graph model.

We deliberately do not adopt the Bayes-optimal nonlinear spectral estimator of Matrix Denoising with Doubly Heteroscedastic Noise: Fundamental Limits and Optimal Spectral Methods as the main method. That paper characterizes the minimum achievable error for a rank-one spike under row and column correlated Gaussian noise. Its estimator assumes the noise covariances are known, and it reports beating whitening, HeteroPCA, OptShrink and ScreeNOT near the weak-recovery threshold. Our setting replaces its known covariances with a plug-in, entrywise, count-driven profile. It stays relevant as a baseline once we feed it our estimated profile.

Baselines

The naive reference point is what most 16S papers do. They apply a centered log-ratio transform with a pseudocount, run plain PCA, and take the first component. A close variant is PCoA on Bray-Curtis distances, the standard beta-diversity ordination (Dimensionality Reduction for Beta Diversity Analysis). Both will be run on exactly the same filtered count tables as our method. They get the same pseudocount and filter search budget, tuned on held-out simulations, so neither is handicapped by poor preprocessing.

The strongest competitor on this specific question is Pearson-residual PCA. It divides centered counts by the square root of the same estimated variance $S_{ij}$, then takes the top singular vectors. It uses the same marginal information we do, so it isolates the one thing we change: the inverse-variance (rather than inverse-standard-deviation) weighting and the threshold-guided filtering. The second strong competitor is the estimator of Zhang and Mondelli, given our plug-in row and column variance estimates in place of known covariances.

For both, the file contains no reported number on 16S data. There is no reported result found on Lacuna for spectral recovery of a planted axis in amplicon count matrices. Zhang and Mondelli report only qualitative superiority over HeteroPCA, OptShrink and ScreeNOT on synthetic doubly heteroscedastic Gaussian matrices, with no exact values in the file. So the comparison must be established here. Beating them means a statistically significant gain in squared overlap $(\hat u^\top u)^2$ and $(\hat v^\top v)^2$ in the window just above the predicted $\theta^\star$, averaged over at least fifty planted replicates per dataset. It also means a lower empirical onset of recovery, measured as the $\theta$ at which overlap exceeds 0.1.

Two ablations isolate the key ingredients. One uses inverse-variance weighting with no weight capping or filtering, which tests the sparse-outlier prediction. The other uses filtering chosen by the conventional fixed-prevalence rule instead of by minimizing $\theta^\star$.

Why not just…?

Why not just run CLR-PCA, as everyone already does? Log-ratio PCA treats every entry as equally noisy after transformation. In reality, a count of 2 in a 3,000-read sample is far noisier than a count of 2,000 in a 60,000-read sample. Mergny et al. show that ignoring a known variance profile moves the detection threshold above the information-theoretic one, even when the signal is recoverable. The planted-axis experiments will measure how large that gap is on real depth and prevalence profiles. If the gap is negligible, then the threshold calculator is the contribution and the estimator is not, and we will report it that way.

Why not just use Pearson residuals, which already divide by the variance? This is the real threat, because it uses exactly our plug-in profile. Our reason to expect a difference is that in variance-profile spiked models the threshold-optimal transform weights by inverse variance, not inverse standard deviation. Whether this matters for 16S profiles is an empirical question. The head-to-head comparison above settles it, and so does a check of whether each method's empirical onset matches its own predicted threshold. If Pearson residuals tie us, we will present the threshold theory and the derived filtering rule as applying to them too.

Why not just use labels with LEfSe or redundancy analysis when the phenotype is known? Supervised tests like LEfSe (Linear Discriminant Analysis Effect Size for Biomarker Discovery) and constrained ordination (Mapping Ecological Communities to Environmental Drivers with Redundancy Analysis) answer a different question: which taxa differ between given groups. Our target is the case where the dysbiotic subset or gradient is not labeled, or labels are coarse. In that case the analyst needs to know whether an unsupervised top axis can be trusted. We will still report LEfSe's taxon ranking against planted $v$ as an upper reference that has access to $u$.

Proposed Experiments

The main testbed is semi-synthetic, built on real amplicon tables. From the Earth Microbiome Project pilot V4 survey of 5,387 samples (The Earth Microbiome Project: The Meeting Report for the 1st International Earth Microbiome Project Conference, Shenzhen, China, June 13th-15th 2011) and from the PIMENTO human gut, soil and marine ENA run collections (PIMENTO: A PrIMEr infereNce TOolkit to facilitate large-scale calling of amplicon sequence variants), we will take environment-homogeneous subsets. We estimate $\hat q$ and $\hat\phi$ from them. We then plant a rank-one log-fold perturbation with chosen sparsity in $u$ and $v$ and resample counts at the observed library sizes. This keeps the real depth distribution, prevalence spectrum and zero pattern while giving ground truth. We sweep $\theta$ and the sample count $n$ by subsampling, and plot overlap curves against the predicted $\theta^\star(n)$.

The decisive check is quantitative. The empirical onset of recovery should fall within the predicted threshold's bootstrap interval across all four biomes. Below the threshold, overlap should stay near zero for every method, including ours. If the onset systematically departs from $\theta^\star$, the plug-in theory is wrong for count data, and we will say so.

The second check targets the sparse-noise prediction. Without weight capping, the top component on a null table with no planted signal should concentrate on a few rare taxa. Its eigenvalue should match the noise outlier from the binned profile, and the threshold-optimal filter should remove it.

The third check tests whether the noise model holds on real libraries rather than simulated ones. The 36-species mock community of Evaluation of 16S rRNA gene sequencing for species and strain-level microbiome analysis has known composition. We will use it to check whether the negative-binomial variance $\mu(1+\phi\mu)$ fits technical replicates. Operon copy number and primer biases from that paper and the EMP report enter our model as fixed multiplicative factors in $q_j$, so they shift the baseline but not the spike. The mock data will show whether that assumption holds.

Finally, as a real-axis test, we will use the EMP host-associated versus free-living split as a known coarse gradient. We will report, as a function of subsampled $n$, whether each method's top axis aligns with it, and whether alignment appears where the calculator says it should.