Large Covariance Estimation for Compositional Data Via Composition-Adjusted Thresholding

Yuanpei Cao, Wei Lin, Hongzhe LiJournal of the American Statistical Association 2016 · arXiv

TL;DR

This paper introduces a new method, Composition-Adjusted Thresholding (COAT), for estimating covariance matrices in high-dimensional compositional data, common in fields like metagenomics. COAT effectively handles the "unit-sum constraint" that plagues standard methods, offering a scalable and theoretically sound approach that outperforms existing techniques.

What Problem Does This Paper Address?

Many real-world datasets, such as the relative abundance of microbial species in a gut sample or the proportions of elements in a rock, are "compositional." This means the data points are non-negative and sum to one, forming a high-dimensional simplex. Traditional statistical methods, especially those dealing with covariance estimation in high dimensions (where the number of variables is large), often fail to produce reliable results for compositional data. This is because the unit-sum constraint introduces spurious correlations, making it difficult to discern true relationships. The paper specifically targets the challenge of estimating the underlying "basis covariance matrix" when the observed data are compositional and the dimensionality is high.

Methods

The core of the COAT method lies in a clever decomposition that relates the observed compositional covariance to an underlying "basis covariance matrix." This basis covariance is assumed to be sparse. The authors leverage the idea that as the dimensionality grows, the compositional effects can be "peeled off," making the basis covariance approximately identifiable. The COAT procedure then involves:

  1. Decomposition: Exploiting a relationship between the compositional covariance and the basis covariance, which involves a rank-2 component and a sparse component.
  2. Approximation: Using the centered log-ratio (CLR) covariance matrix as a proxy for the basis covariance. This is a key insight, as it allows for a more tractable estimation.
  3. Thresholding: Applying an adaptive thresholding procedure to the estimated CLR covariance matrix. This is a standard technique in high-dimensional statistics for identifying sparse structures.

The method is described as "optimization-free" and scalable, meaning it doesn't require computationally intensive optimization steps and can handle large datasets.

Datasets

The paper uses simulated data for its theoretical analysis and performance evaluation. For a real-world application, they analyze a human gut microbiome dataset previously described by Wu et al. (2011). This dataset consists of 16S rRNA gene sequencing data from 98 healthy individuals, with taxonomic assignments yielding 87 genera. The authors divide this dataset into "lean" and "obese" groups based on BMI and focus on 40 bacterial genera that appear in at least four samples in each group.

Key Insights and Ideas

  • "Blessing of Dimensionality" for Identifiability: Counterintuitively, the high dimensionality of compositional data, under certain sparsity assumptions, actually helps in identifying the underlying basis covariance structure.
  • CLR as a Proxy: The centered log-ratio (CLR) covariance matrix serves as a crucial and effective proxy for the true basis covariance, simplifying the estimation problem.
  • Composition-Adjusted Thresholding: The COAT method effectively "removes the coat" of compositional effects, allowing for more accurate estimation of the underlying sparse covariance.
  • Tradeoff in Taxonomic Levels: The theoretical results suggest a tradeoff: analyzing data at lower taxonomic levels (fewer variables) might lead to more interpretable results, while higher levels (more variables) are needed for accurate estimation of the underlying basis structure.

Key Results

  • Theoretical Guarantees: The paper provides rigorous theoretical guarantees for the COAT estimator, including rates of convergence under the spectral norm and support recovery properties.
  • Outperforms Naive Methods: Simulation studies show that COAT significantly outperforms naive thresholding estimators that ignore the compositional nature of the data, especially in settings with skewed distributions (common in microbiome data).
  • Microbiome Application: Applying COAT to the gut microbiome data reveals distinct correlation network structures between lean and obese individuals, offering biologically interpretable insights into microbial community ecology. The COAT method also produced more stable correlation networks compared to naive methods.

Limitations and Future Work

  • Approximation Error: The COAT estimator's rate of convergence includes an "approximation error" term arising from using the CLR covariance as a proxy for the basis covariance. While this error diminishes with increasing dimensionality, it's a factor to consider.
  • Zero Counts: The current method transforms zero counts by replacing them with 0.5. The authors suggest that a worthwhile extension would be to directly handle sampling and structural zeros, which is a complex issue due to identifiability and computational challenges.
  • Joint Optimization: The paper proposes extending the methodology to a joint optimization procedure based on the decomposition, which might offer improved performance over the one-step COAT approximation.

Summary updated

Abstract

High-dimensional compositional data arise naturally in many applications such as metagenomic data analysis. The observed data lie in a high-dimensional simplex, and conventional statistical methods often fail to produce sensible results due to the unit-sum constraint. In this article, we address the problem of covariance estimation for high-dimensional compositional data and introduce a composition-adjusted thresholding (COAT) method under the assumption that the basis covariance matrix is sparse. Our method is based on a decomposition relating the compositional covariance to the basis covariance, which is approximately identifiable as the dimensionality tends to infinity. The resulting procedure can be viewed as thresholding the sample centered log-ratio covariance matrix and hence is scalable for large covariance matrices. We rigorously characterize the identifiability of the covariance parameters, derive rates of convergence under the spectral norm, and provide theoretical guarantees on support recovery. Simulation studies demonstrate that the COAT estimator outperforms some existing optimization-based estimators. We apply the proposed method to the analysis of a microbiome dataset to understand the dependence structure among bacterial taxa in the human gut.
BibTeX
@article{cao2016large,
  author = {Yuanpei Cao and Wei Lin and Hongzhe Li},
  title = {{Large Covariance Estimation for Compositional Data Via Composition-Adjusted Thresholding}},
  journal = {Journal of the American Statistical Association},
  year = {2016},
}
Figure