HePPCAT: Probabilistic PCA for Data With Heteroscedastic Noise

David Hong, Kyle Gilman, Laura Balzano, Jeffrey A. FesslerIEEE TSP 2021 · OpenReview

Abstract

Principal component analysis (PCA) is a classical and ubiquitous method for reducing data dimensionality, but it is suboptimal for heterogeneous data that are increasingly common in modern applications. PCA treats all samples uniformly so degrades when the noise is heteroscedastic across samples, as occurs, e.g., when samples come from sources of heterogeneous quality. This paper develops a probabilistic PCA variant that estimates and accounts for this heterogeneity by incorporating it in the statistical model. Unlike in the homoscedastic setting, the resulting nonconvex optimization problem is not seemingly solved by singular value decomposition. This paper develops a heteroscedastic probabilistic PCA technique (HePPCAT) that uses efficient alternating maximization algorithms to jointly estimate both the underlying factors and the unknown noise variances. Simulation experiments illustrate the comparative speed of the algorithms, the benefit of accounting for heteroscedasticity, and the seemingly favorable optimization landscape of this problem. Real data experiments on environmental air quality data show that HePPCAT can give a better PCA estimate than techniques that do not account for heteroscedasticity.

Introduction

Principal Component Analysis (PCA) serves as a foundational tool in data science for dimensionality reduction and feature extraction. However, its standard formulation relies on the assumption of homoscedasticity—that all data points are subject to the same level of noise. In contemporary "big data" contexts, datasets are frequently aggregated from diverse sources with varying levels of precision, such as combining high-grade scientific sensors with low-cost consumer devices. In these heteroscedastic scenarios, standard PCA can be severely biased by noisy samples, which "drown out" the signal from higher-quality observations.

The HePPCAT Framework

To address this limitation, the authors propose HePPCAT (Heteroscedastic Probabilistic PCA Technique). HePPCAT extends the probabilistic PCA (PPCA) framework by explicitly modeling sample-specific noise variances. By incorporating these variances into the statistical model, the method can effectively "down-weight" noisier samples during the estimation of the underlying principal components.

Unlike standard PCA, which can be solved globally via Singular Value Decomposition (SVD), the introduction of heteroscedastic noise leads to a non-convex optimization problem. The authors tackle this using a Block Coordinate Ascent algorithm that alternates between two primary updates:

  1. Factor Estimation ($F$): An Expectation-Maximization (EM) step estimates the underlying factors while keeping noise estimates fixed.
  2. Noise Variance Estimation ($v$): The authors evaluate several strategies for updating noise levels, including high-accuracy root-finding and more efficient Quadratic and Cubic Solvable Minorizers. These minorizers provide closed-form-style updates that significantly accelerate the algorithm.

Experimental Evaluation

The performance of HePPCAT was validated through both synthetic simulations and real-world applications:

  • Synthetic Simulations: These allowed for precise error measurement against known ground-truth factors and noise levels across various data realizations.
  • Environmental Air Quality Data: The model was tested on PM2.5 readings from Central California, merging data from 46 high-precision EPA sensors with 460 low-cost "PurpleAir" IoT sensors. This dataset exemplifies the challenge of heterogeneous data quality in modern environmental monitoring.

Results and Discussion

The experimental results highlight several key advantages of the HePPCAT approach.

Comparison of PCA and HePPCAT on synthetic data with heteroscedastic noise.

Accuracy and Robustness

In simulations, HePPCAT consistently outperformed standard PCA. As shown in the figure above, standard PCA (purple line) is significantly pulled toward the spread of the noisy samples (blue dots). In contrast, HePPCAT (brown line) remains closely aligned with the true signal (green line), effectively matching the performance of "Oracle" methods that have prior knowledge of the noise levels. This robustness is maintained regardless of whether clean samples outnumber noisy ones (a) or vice versa (b).

Optimization Landscape

Despite the non-convex nature of the objective function, the authors observed a remarkably "well-behaved" optimization landscape. In empirical tests, the algorithm converged to the same global maximum regardless of initialization, suggesting that local optima may not be a significant hurdle for this specific problem class.

Real-World Performance

On the air quality dataset, HePPCAT achieved a lower Normalized Root Mean-Squared Error (NRMSE) on test data compared to standard PCA. This indicates that by learning to account for the higher noise in crowdsourced sensors, HePPCAT recovers a more accurate representation of the underlying air quality patterns.

Conclusion

HePPCAT provides a robust alternative to standard PCA for datasets characterized by varying sample quality. While the current work focuses on sample-wise heteroscedasticity, future research may extend this to feature-wise noise variations or incorporate automated clustering to group samples with similar noise characteristics. Additionally, addressing the slight downward bias observed in noise estimation remains an area for further refinement.

Summary updated

Abstract

Principal component analysis (PCA) is a classical and ubiquitous method for reducing data dimensionality, but it is suboptimal for heterogeneous data that are increasingly common in modern applications. PCA treats all samples uniformly so degrades when the noise is heteroscedastic across samples, as occurs, e.g., when samples come from sources of heterogeneous quality. This paper develops a probabilistic PCA variant that estimates and accounts for this heterogeneity by incorporating it in the statistical model. Unlike in the homoscedastic setting, the resulting nonconvex optimization problem is not seemingly solved by singular value decomposition. This paper develops a heteroscedastic probabilistic PCA technique (HePPCAT) that uses efficient alternating maximization algorithms to jointly estimate both the underlying factors and the unknown noise variances. Simulation experiments illustrate the comparative speed of the algorithms, the benefit of accounting for heteroscedasticity, and the seemingly favorable optimization landscape of this problem. Real data experiments on environmental air quality data show that HePPCAT can give a better PCA estimate than techniques that do not account for heteroscedasticity.
BibTeX
@article{hong2021heppcat,
  author = {David Hong and Kyle Gilman and Laura Balzano and Jeffrey A. Fessler},
  title = {{HePPCAT: Probabilistic PCA for Data With Heteroscedastic Noise}},
  journal = {IEEE Trans. Signal Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=A4ow7kFAkRNR},
}
Figure