Joint Mean-Covariance Estimation via the Horseshoe with an Application in Genomic Data Analysis

Yunfan Li, Jyotishka Datta, Bruce Α. Craig, Anindya BhadraarXiv 2019 · DOI · arXiv

Abstract

Seemingly unrelated regression is a natural framework for regressing multiple correlated responses on multiple predictors. The model is very flexible, with multiple linear regression and covariance selection models being special cases. However, its practical deployment in genomic data analysis under a Bayesian framework is limited due to both statistical and computational challenges. The statistical challenge is that one needs to infer both the mean vector and the inverse covariance matrix, a problem inherently more complex than separately estimating each. The computational challenge is due to the dimensionality of the parameter space that routinely exceeds the sample size. We propose the use of horseshoe priors on both the mean vector and the inverse covariance matrix. This prior has demonstrated excellent performance when estimating a mean vector or inverse covariance matrix separately. The current work shows these advantages are also present when addressing both simultaneously. A full Bayesian treatment is proposed, with a sampling algorithm that is linear in the number of predictors. MATLAB code implementing the algorithm is freely available from github at https://github.com/liyf1988/HS_GHS. Extensive performance comparisons are provided with both frequentist and Bayesian alternatives, and both estimation and prediction performances are verified on a genomic data set.

Introduction

In high-dimensional data analysis—particularly in fields like genomics, finance, and chemometrics—researchers frequently encounter the "p > n" problem, where the number of predictors ($p$) significantly exceeds the number of samples ($n$). Furthermore, the responses being measured (e.g., gene expression levels) are often intrinsically correlated.

Seemingly Unrelated Regression (SUR) provides a robust framework for addressing these dependencies by regressing multiple correlated responses on multiple predictors simultaneously. However, joint estimation of the regression coefficients and the precision matrix (the inverse of the covariance matrix) presents significant statistical and computational hurdles. This paper introduces HS-GHS, a fully Bayesian framework that utilizes Horseshoe priors to achieve sparse estimation in both the mean and covariance structures, offering a scalable solution for complex genomic datasets.

The HS-GHS Framework

The proposed model, HS-GHS, integrates the Horseshoe (HS) prior for regression coefficients with the Graphical Horseshoe (GHS) prior for the precision matrix.

Global-Local Shrinkage

The Horseshoe prior is a "global-local" shrinkage prior characterized by a tall "spike" at zero and heavy tails. This structure allows the model to effectively push irrelevant variables toward zero (sparsity) while ensuring that truly significant predictors are not over-shrunk. By applying this to both the regression coefficients ($B$) and the precision matrix ($\Omega$), the model can simultaneously map genetic influences and discover underlying biological networks.

Joint Estimation

Unlike traditional methods that might estimate the mean and covariance separately, HS-GHS performs joint estimation. In the context of the precision matrix, the zeros represent conditional independence between variables, which effectively maps the edges of a biological network. This joint approach is statistically more rigorous than separate estimation, as it accounts for the interplay between the predictors and the response correlations.

Computational Efficiency

A primary barrier to adopting Bayesian SUR models has been the computational cost of Markov Chain Monte Carlo (MCMC) sampling. The authors address this by implementing a "fast sampling" algorithm. This approach reduces the computational complexity to be linear in the number of predictors ($p$). While the complexity remains cubic in the number of responses ($q$), this linear scaling in $p$ is a significant advancement for genomic studies where the number of genetic markers is typically the primary bottleneck.

Empirical Evaluation

Simulated Data

The model was tested across various simulated settings, including different precision matrix structures (such as AR1 and Clique structures) and varying coefficient sizes. HS-GHS consistently demonstrated superior performance in identifying true signals while maintaining a low false-positive rate.

Yeast eQTL Analysis

The practical utility of HS-GHS was demonstrated using a yeast expression quantitative trait loci (eQTL) dataset involving 112 yeast segregants, 49 genes in the MAPK signaling pathway, and 172 genetic markers.

  • Prediction Accuracy: HS-GHS achieved an average prediction $R^2$ of 0.2771, significantly outperforming frequentist alternatives like MRCE (0.0063) and other Bayesian methods like CAPME (0.1327).
  • Biological Insights: The model successfully identified "eQTL hotspots" on chromosomes 3, 5, and 14, which act as master regulators for multiple genes.
  • Network Recovery: The inferred gene network recovered known structures within the yeast MAPK pathway, correctly linking genes associated with mating, cell wall remodeling, and osmolyte synthesis.

Conclusion

HS-GHS represents a significant step forward in high-dimensional genomic analysis. By leveraging the unique properties of the Horseshoe prior, the model provides high precision and automated sparsity without the need for manual tuning or cross-validation. While the cubic scaling in the number of responses ($q$) suggests that pre-screening may still be necessary for extremely large gene sets, the linear scaling in $p$ makes it a powerful tool for modern genetic mapping.

Implementation: The MATLAB code for HS-GHS is available at https://github.com/liyf1988/HS_GHS.

Summary updated

Abstract

Seemingly unrelated regression is a natural framework for regressing multiple correlated responses on multiple predictors. The model is very flexible, with multiple linear regression and covariance selection models being special cases. However, its practical deployment in genomic data analysis under a Bayesian framework is limited due to both statistical and computational challenges. The statistical challenge is that one needs to infer both the mean vector and the inverse covariance matrix, a problem inherently more complex than separately estimating each. The computational challenge is due to the dimensionality of the parameter space that routinely exceeds the sample size. We propose the use of horseshoe priors on both the mean vector and the inverse covariance matrix. This prior has demonstrated excellent performance when estimating a mean vector or inverse covariance matrix separately. The current work shows these advantages are also present when addressing both simultaneously. A full Bayesian treatment is proposed, with a sampling algorithm that is linear in the number of predictors. MATLAB code implementing the algorithm is freely available from github at https://github.com/liyf1988/HS_GHS. Extensive performance comparisons are provided with both frequentist and Bayesian alternatives, and both estimation and prediction performances are verified on a genomic data set.
BibTeX
@misc{li2019joint,
  author = {Yunfan Li and Jyotishka Datta and Bruce Α. Craig and Anindya Bhadra},
  title = {{Joint Mean-Covariance Estimation via the Horseshoe with an Application in Genomic Data Analysis}},
  year = {2019},
  eprint = {1903.06768},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.06768},
}
Figure