Generalized probabilistic principal component analysis of correlated data

Mengyang Gu, Weining ShenJMLR 2020

TL;DR

This paper introduces Generalized Probabilistic Principal Component Analysis (GPPCA), a novel method for dimension reduction that accounts for correlations between latent factors. GPPCA offers a more accurate and computationally efficient way to model complex, correlated data compared to traditional PCA and its probabilistic extensions.

What Problem Does This Paper Address?

Principal Component Analysis (PCA) is a cornerstone technique for reducing the dimensionality of data by finding uncorrelated principal components. While effective, standard PCA lacks a probabilistic foundation. Probabilistic PCA (PPCA) addresses this by modeling data within a probabilistic framework, but it makes a restrictive assumption that latent factors are independent. This assumption is often unrealistic in real-world scenarios like modeling multiple time series, spatial processes, or functional data, where these latent factors are inherently correlated. Existing methods to handle correlated factors can be computationally intensive or lack a closed-form solution. This paper aims to bridge this gap by developing a probabilistic PCA that explicitly models correlated latent factors, offering both theoretical elegance and practical efficiency.

Methods

The core contribution is the development of Generalized Probabilistic Principal Component Analysis (GPPCA). GPPCA extends PPCA by modeling each latent factor using a Gaussian Process (GP), allowing for correlations between these factors. The paper derives a closed-form maximum marginal likelihood estimator (MMLE) for the factor loading matrix, which is a key advancement. This is achieved by carefully deriving the precision matrix of the marginal likelihood.

Key methodological aspects include:

  • Gaussian Process Latent Factors: Instead of independent normal distributions, each latent factor is modeled as a zero-mean Gaussian Process. This allows for flexible modeling of correlations between factors.
  • Closed-Form MMLE: The paper provides a computationally efficient closed-form solution for the factor loading matrix under certain conditions (e.g., shared covariance functions). When covariance functions differ, it formulates an optimization problem on the Stiefel manifold, for which numerical solutions are provided.
  • Marginalization of Covariates: GPPCA can incorporate covariates into the mean structure of the model and marginalize them out, providing a fully probabilistic analysis without increasing computational complexity. This is a significant improvement over methods that simply subtract the mean.
  • Kernel Methods: The paper uses kernel functions to parameterize the covariance structure of the GPs, demonstrating flexibility with common kernels like the Matérn kernel.

Datasets

The paper evaluates GPPCA on simulated data and two real-world datasets:

  • Simulated Data: Used to compare GPPCA against PCA, LY1, and LY5 under various scenarios, including shared and different covariance matrices for factors, and deterministic factors.
  • DIAMOND Simulator Data: A benchmark dataset for emulating computer models with multivariate outputs. This dataset involves modeling the number of casualties after simulated earthquake and volcanic eruptions, with 13 input variables and multivariate outputs.
  • Gridded Temperature Data: A dataset from NOAA containing global gridded temperature anomalies. This data is used for interpolation tasks, with missing values being a common challenge.

Key Insights and Ideas

  • Correlated Factors Matter: The paper strongly demonstrates that explicitly modeling correlations between latent factors (via GPs) leads to more accurate estimation of the factor loading matrix and improved predictive performance, especially when data exhibits strong correlations.
  • Computational Efficiency: Deriving a closed-form solution for the marginal likelihood and factor loadings is a significant practical advantage, making GPPCA scalable to larger datasets.
  • Probabilistic Mean Structure: Incorporating covariates into the mean and marginalizing them out provides a more robust and uncertainty-aware analysis compared to simple mean subtraction.
  • Flexibility of GPs: The use of Gaussian Processes for latent factors allows GPPCA to adapt to various covariance structures, making it a versatile tool.

Key Results

  • Improved Estimation Accuracy: In simulated examples, GPPCA consistently outperforms PCA, LY1, and LY5 in estimating the factor loading matrix, particularly when factors are correlated or noise levels are high. The largest principal angle between true and estimated subspaces is significantly smaller for GPPCA.
  • Superior Prediction Performance: For the DIAMOND simulator data, GPPCA achieves the smallest out-of-sample RMSE and provides more precise predictive intervals compared to Independent GP and Multi GP models.
  • Accurate Interpolation: In the gridded temperature anomaly dataset, GPPCA demonstrates more precise interpolation than spatio-temporal models, especially in regions with rapid anomaly changes.
  • Scalability: The closed-form expression for the marginal likelihood leads to computational complexity that is linear in the number of output variables, making it efficient.

Limitations and Future Work

  • Number of Latent Factors: The current GPPCA framework assumes the number of latent factors ($d$) is known. The authors suggest developing a consistent method to identify this number as a future direction.
  • Parameter Uncertainty: While GPPCA provides MMLEs, the uncertainty in parameter estimation is not fully captured in the predictive distributions. A full Bayesian approach is proposed to address this.
  • Optimization Convergence: For cases with different covariance functions, numerical optimization on the Stiefel manifold is used. Exploring faster algorithms and analyzing convergence rates for this optimization is suggested.
  • Spatial Distance: The current GPPCA formulation does not explicitly use spatial distance information, making it suitable for general matrix completion tasks but potentially limiting for purely spatial interpolation where distance is crucial.
  • Missing Data: Extending GPPCA to handle missing values in the observation matrix is identified as a promising area for future research, potentially using Expectation-Maximization or MCMC.

Summary updated

BibTeX
@article{gu2020generalized,
  author = {Mengyang Gu and Weining Shen},
  title = {{Generalized probabilistic principal component analysis of correlated data}},
  journal = {JMLR},
  year = {2020},
}
Figure