Bayesian group factor analysis with structured sparsity

Shiwen Zhao, Chuan Gao, Sayan Mukherjee, Barbara E. EngelhardtJMLR 2016 · OpenReview

TL;DR

This paper introduces BASS, a novel Bayesian group factor analysis model that uses a hierarchical prior to simultaneously capture both shared and observation-specific patterns in multiple datasets. BASS excels at uncovering both sparse and dense latent factors, making it robust for analyzing complex, high-dimensional data like gene expression and text.

What Problem Does This Paper Address?

High-dimensional data, where the number of features ($p$) far exceeds the number of samples ($n$), presents a significant challenge for traditional statistical methods. Latent factor models are powerful tools for uncovering underlying low-dimensional structure in such data. However, existing methods often struggle to simultaneously:

  • Handle multiple related datasets (observations): Many methods are designed for a single dataset, making it difficult to leverage information across related sources.
  • Distinguish between shared and unique patterns: It's crucial to identify factors that influence all datasets versus those specific to individual ones.
  • Recover both sparse and dense signals: Some patterns might affect only a few features (sparse), while others might influence many (dense). Existing models often favor one or the other.
  • Achieve interpretable results: Unsupervised discovery of latent factors that correspond to meaningful biological or textual structures is a key goal.

The authors aim to address these limitations by developing a flexible Bayesian framework that can handle multiple observations and explicitly model structured sparsity.

Methods

The core contribution of this paper is the Bayesian group factor Analysis with Structured Sparsity (BASS) model. Key methodological aspects include:

  • Hierarchical Bayesian Prior: BASS employs a novel, three-level hierarchical prior based on the three-parameter beta (TPB) distribution. This prior is applied to the factor loading matrix and is designed to induce:
  • Global shrinkage: To remove irrelevant factors.
  • Factor-specific shrinkage: To decouple latent spaces from arbitrary subsets of observations (column-wise sparsity).
  • Local shrinkage: To encourage element-wise sparsity within loading columns, leading to interpretable factors.
  • Mixture of Sparse and Dense Factors: The model explicitly allows for both sparse (affecting few features) and dense (affecting many features) latent factors, which is crucial for real-world data where both types of signals are common.
  • Parameter-Expanded Expectation-Maximization (PX-EM): To efficiently estimate model parameters, the authors develop a PX-EM algorithm. This method is designed to be more robust to initialization and to navigate the complex posterior landscape of the model, offering improved convergence properties compared to standard EM.

Datasets

The authors validate their BASS model on both simulated and real-world datasets:

  • Simulated Data:
    • Paired Observations (CCA): Two datasets ($p_1=100, p_2=120$) with varying sample sizes ($n \in \{20, 30, 40, 50\}$) were used to simulate scenarios with sparse, dense, and mixed sparse/dense factors.
  • Four Observations (GFA): Four datasets with varying dimensions were used to test the model's performance with more than two observations.
  • Ten Observations (GFA): Ten datasets were used to evaluate scalability and performance with a larger number of coupled observations.
  • Real-World Data:
    • Mulan Library: Ten benchmark datasets for multi-label prediction were used to evaluate prediction accuracy.
  • Cholesterol and Pharmacogenomic (CAP) Study: Gene expression data from 480 lymphoblastoid cell lines under two conditions were analyzed to detect condition-specific sparse covariance structures.
  • 20 Newsgroups Data: A text dataset was used for multiclass classification.

Key Insights and Ideas

  • Structured Sparsity is Key: The hierarchical TPB prior is the core innovation, enabling BASS to simultaneously learn which factors are important (global), which observations they relate to (factor-specific), and which features they influence (element-wise). This multi-level shrinkage is crucial for disentangling complex relationships.
  • Dense and Sparse Factors Coexist: Real-world data often contains both broad, dense signals (e.g., batch effects) and specific, sparse signals (e.g., gene-gene interactions). BASS's ability to model both is a significant advantage.
  • PX-EM for Robustness: The PX-EM algorithm is essential for making the model practical. It helps overcome the common issue of EM getting stuck in local optima, leading to more reliable results.
  • Interpretability through Sparsity: By inducing sparsity, BASS facilitates the interpretation of latent factors. Sparse loadings highlight specific features that contribute to a factor, making it easier to understand the underlying biological or textual meaning.

Key Results

  • Superior Recovery of Latent Factors: Across simulations, BASS consistently outperformed other methods (GFA, sGFA, SCCA, JFA, RCCA, MRD-lin) in recovering the true sparse and dense latent factors, as measured by stability indices (SSI and DSI).
  • High Prediction Accuracy: BASS achieved the best prediction accuracy on several datasets, particularly those with continuous responses or complex interactions, demonstrating its ability to generalize well.
  • Scalability: The PX-EM algorithm allows BASS to scale to large datasets, as evidenced by its application to gene expression and text data with thousands of features and observations.
  • Biologically Meaningful Discoveries: In the CAP gene expression data, BASS identified condition-specific co-expression networks, highlighting genes uniquely regulated by different experimental conditions.
  • Text Classification Performance: BASS achieved competitive accuracy in classifying documents into newsgroups, demonstrating its utility in text analysis.

Limitations and Future Work

  • Initialization Sensitivity: While PX-EM improves robustness, the authors note that EM can still occasionally get stuck in poor local optima, especially with a small number of PX-EM iterations.
  • Computational Complexity: Although PX-EM is efficient, the computational complexity can still be demanding for extremely large datasets, particularly when the number of factors ($k$) is large.
  • Identifiability: Like other factor analysis models, BASS faces challenges with rotational invariance and label switching, which are addressed through post-processing steps.
  • Future Work: The authors suggest extending BASS to non-linear factor models, incorporating structured Gaussian process kernels, handling missing data, and developing methods to include domain-specific structure across samples or features. They also highlight the potential for further research into scaling these methods for even larger genomics and neuroscience datasets.

Summary updated

Abstract

Latent factor models are the canonical statistical tool for exploratory analyses of low-dimensional linear structure for a matrix of $p$ features across $n$ samples. We develop a structured Bayesian group factor analysis model that extends the factor model to multiple coupled observation matrices; in the case of two observations, this reduces to a Bayesian model of canonical correlation analysis. Here, we carefully define a structured Bayesian prior that encourages both element-wise and column-wise shrinkage and leads to desirable behavior on high- dimensional data. In particular, our model puts a structured prior on the joint factor loading matrix, regularizing at three levels, which enables element-wise sparsity and unsupervised recovery of latent factors corresponding to structured variance across arbitrary subsets of the observations. In addition, our structured prior allows for both dense and sparse latent factors so that covariation among either all features or only a subset of features can be recovered. We use fast parameter-expanded expectation-maximization for parameter estimation in this model. We validate our method on simulated data with substantial structure. We show results of our method applied to three high- dimensional data sets, comparing results against a number of state-of-the-art approaches. These results illustrate useful properties of our model, including i) recovering sparse signal in the presence of dense effects; ii) the ability to scale naturally to large numbers of observations; iii) flexible observation- and factor-specific regularization to recover factors with a wide variety of sparsity levels and percentage of variance explained; and iv) tractable inference that scales to modern genomic and text data sizes.
BibTeX
@article{zhao2016bayesian,
  author = {Shiwen Zhao and Chuan Gao and Sayan Mukherjee and Barbara E. Engelhardt},
  title = {{Bayesian group factor analysis with structured sparsity}},
  journal = {J. Mach. Learn. Res.},
  year = {2016},
  url = {https://openreview.net/forum?id=Cv6W3tsGNX},
}
Figure