Robust Covariance Estimation for High-dimensional Compositional Data with Application to Microbial Communities Analysis

Yong He, Pengfei Liu, Xinsheng Zhang, Zhou WangarXiv 2020 · DOI · arXiv

Abstract

Microbial communities analysis is drawing growing attention due to the rapid development of high-throughput sequencing techniques. The observed data is typically high-dimensional, compositional, and often leptokurtic and highly skewed due to overly abundant taxa. These characteristics make conventional correlation analysis infeasible. We address these challenges by proposing RCEC (Robust Covariance Estimator for Compositional data). Assuming the basis covariance matrix lies in a class of sparse matrices, we adopt the centered log-ratio (clr) covariance matrix as a proxy. We construct a Median-of-Means (MOM) estimator for the clr covariance matrix and propose an adaptive thresholding procedure. Under a weak finite fourth moment condition, we derive the optimal rate of convergence under the spectral norm and provide theoretical guarantees on support recovery. Simulation studies and application to a human gut microbiome dataset demonstrate the advantages of RCEC over state-of-the-art methods.

Introduction

In microbiome research, understanding bacterial interactions—who thrives together and who competes—is a primary goal. However, high-throughput sequencing data presents significant statistical hurdles: it is high-dimensional, compositional (summing to 100%), and frequently heavy-tailed or skewed. Standard correlation analysis is mathematically invalid for compositional data because the unit-sum constraint induces spurious correlations. Furthermore, existing tools often assume sub-Gaussian distributions, failing when faced with the "messy" nature of real-world microbial data where a few species dominate and outliers are common.

Methodology

The proposed RCEC method utilizes a two-step robust estimation procedure:

  1. Median-of-Means (MOM) Estimator: To mitigate the impact of outliers and heavy-tailedness, the data is divided into $M$ blocks. The mean is calculated for each block, and the final estimate is the median of these means. This provides a robust alternative to the standard sample mean.
  2. Adaptive Thresholding: To handle high dimensionality ($p \gg n$), a thresholding procedure is applied to enforce sparsity. Unlike fixed-cutoff methods, this threshold is adaptive, adjusting based on the variability of individual entries in the covariance matrix.

The method uses the centered log-ratio (clr) covariance matrix as a proxy for the unobservable basis covariance of absolute abundances. Theoretically, RCEC requires only a finite fourth moment condition, a much weaker assumption than the sub-Gaussianity required by previous methods.

Experimental Results

The researchers evaluated RCEC using synthetic simulations and a human gut microbiome dataset consisting of 40 bacterial genera from 63 lean and 35 obese subjects.

Simulation Performance

In scenarios mimicking heavy-tailed and contaminated data, RCEC consistently outperformed competitors such as COAT, CCLasso, SparCC, and REBACCA.

Comparison of estimation error (spectral norm loss) across different methods.

As shown in the figure above, RCEC achieves the lowest spectral norm loss, indicating superior accuracy and robustness compared to state-of-the-art methods, particularly SparCC which shows high variability and error in these settings.

Real-World Application

In the human gut analysis, RCEC demonstrated higher network stability (0.712) compared to the COAT method (0.584). The analysis revealed that the "lean" group possesses a more complex, modular interaction network, whereas the "obese" group's network is characterized by more competitive (negative) interactions, suggesting obesity might be associated with a less stable microbial community.

Discussion

While RCEC offers significant improvements, challenges remain. The resulting covariance matrix is not guaranteed to be positive-definite, a common issue in high-dimensional thresholding. Additionally, while adaptive thresholding is powerful, the cross-validation required for parameter tuning can be computationally intensive for very high-dimensional datasets. Future work may explore alternative robust estimators like Huber’s M-estimator.

The R script for implementing RCEC is available at https://github.com/heyongstat/RCEC.

Summary updated

Abstract

Microbial communities analysis is drawing growing attention due to the rapid development of high-throughput sequencing techniques nowadays. The observed data has the following typical characteristics: it is high-dimensional, compositional (lying in a simplex) and even would be leptokurtic and highly skewed due to the existence of overly abundant taxa, which makes the conventional correlation analysis infeasible to study the co-occurrence and co-exclusion relationship between microbial taxa. In this article, we address the challenges of covariance estimation for this kind of data. Assuming the basis covariance matrix lying in a well-recognized class of sparse covariance matrices, we adopt a proxy matrix known as centered log-ratio covariance matrix in the literature, which is approximately indistinguishable from the real basis covariance matrix as the dimensionality tends to infinity. We construct a Median-of-Means (MOM) estimator for the centered log-ratio covariance matrix and propose a thresholding procedure that is adaptive to the variability of individual entries. By imposing a much weaker finite fourth moment condition compared with the sub-Gaussianity condition in the literature, we derive the optimal rate of convergence under the spectral norm. In addition, we also provide theoretical guarantee on support recovery. The adaptive thresholding procedure of the MOM estimator is easy to implement and gains robustness when outliers or heavy-tailedness exist. Thorough simulation studies are conducted to show the advantages of the proposed procedure over some state-of-the-arts methods. At last, we apply the proposed method to analyze a microbiome dataset in human gut. The R script for implementing the method is available at https://github.com/heyongstat/RCEC.
BibTeX
@misc{he2020robust,
  author = {Yong He and Pengfei Liu and Xinsheng Zhang and Zhou Wang},
  title = {{Robust Covariance Estimation for High-dimensional Compositional Data with Application to Microbial Communities Analysis}},
  year = {2020},
  eprint = {2004.09018},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.09018},
}
Figure