Abstract
Microbial communities analysis is drawing growing attention due to the rapid development of high-throughput sequencing techniques. The observed data is typically high-dimensional, compositional, and often leptokurtic and highly skewed due to overly abundant taxa. These characteristics make conventional correlation analysis infeasible. We address these challenges by proposing RCEC (Robust Covariance Estimator for Compositional data). Assuming the basis covariance matrix lies in a class of sparse matrices, we adopt the centered log-ratio (clr) covariance matrix as a proxy. We construct a Median-of-Means (MOM) estimator for the clr covariance matrix and propose an adaptive thresholding procedure. Under a weak finite fourth moment condition, we derive the optimal rate of convergence under the spectral norm and provide theoretical guarantees on support recovery. Simulation studies and application to a human gut microbiome dataset demonstrate the advantages of RCEC over state-of-the-art methods.
Introduction
In microbiome research, understanding bacterial interactions—who thrives together and who competes—is a primary goal. However, high-throughput sequencing data presents significant statistical hurdles: it is high-dimensional, compositional (summing to 100%), and frequently heavy-tailed or skewed. Standard correlation analysis is mathematically invalid for compositional data because the unit-sum constraint induces spurious correlations. Furthermore, existing tools often assume sub-Gaussian distributions, failing when faced with the "messy" nature of real-world microbial data where a few species dominate and outliers are common.
Methodology
The proposed RCEC method utilizes a two-step robust estimation procedure:
- Median-of-Means (MOM) Estimator: To mitigate the impact of outliers and heavy-tailedness, the data is divided into $M$ blocks. The mean is calculated for each block, and the final estimate is the median of these means. This provides a robust alternative to the standard sample mean.
- Adaptive Thresholding: To handle high dimensionality ($p \gg n$), a thresholding procedure is applied to enforce sparsity. Unlike fixed-cutoff methods, this threshold is adaptive, adjusting based on the variability of individual entries in the covariance matrix.
The method uses the centered log-ratio (clr) covariance matrix as a proxy for the unobservable basis covariance of absolute abundances. Theoretically, RCEC requires only a finite fourth moment condition, a much weaker assumption than the sub-Gaussianity required by previous methods.
Experimental Results
The researchers evaluated RCEC using synthetic simulations and a human gut microbiome dataset consisting of 40 bacterial genera from 63 lean and 35 obese subjects.
Simulation Performance
In scenarios mimicking heavy-tailed and contaminated data, RCEC consistently outperformed competitors such as COAT, CCLasso, SparCC, and REBACCA.
As shown in the figure above, RCEC achieves the lowest spectral norm loss, indicating superior accuracy and robustness compared to state-of-the-art methods, particularly SparCC which shows high variability and error in these settings.
Real-World Application
In the human gut analysis, RCEC demonstrated higher network stability (0.712) compared to the COAT method (0.584). The analysis revealed that the "lean" group possesses a more complex, modular interaction network, whereas the "obese" group's network is characterized by more competitive (negative) interactions, suggesting obesity might be associated with a less stable microbial community.
Discussion
While RCEC offers significant improvements, challenges remain. The resulting covariance matrix is not guaranteed to be positive-definite, a common issue in high-dimensional thresholding. Additionally, while adaptive thresholding is powerful, the cross-validation required for parameter tuning can be computationally intensive for very high-dimensional datasets. Future work may explore alternative robust estimators like Huber’s M-estimator.
The R script for implementing RCEC is available at https://github.com/heyongstat/RCEC.
Summary updated