HeMPPCAT: Mixtures of Probabilistic Principal Component Analysers for Data with Heteroscedastic Noise

Alec S. Xu, Laura Balzano, Jeffrey A. Fessler

Abstract

Mixtures of probabilistic principal component analysis (MPPCA) is a well-known mixture model extension of principal component analysis (PCA). Similar to PCA, MPPCA assumes the data samples in each mixture contain homoscedastic noise. However, datasets with heterogeneous noise across samples are becoming increasingly common, as larger datasets are generated by collecting samples from several sources with varying noise profiles. The performance of MPPCA is suboptimal for data with heteroscedastic noise across samples. This paper proposes a heteroscedastic mixtures of probabilistic PCA technique (HeMPPCAT) that uses a generalized expectation-maximization (GEM) algorithm to jointly estimate the unknown underlying factors, means, and noise variances under a heteroscedastic noise setting. Simulation results illustrate the improved factor estimates and clustering accuracies of HeMPPCAT compared to MPPCA.

Introduction

In high-dimensional data analysis, Principal Component Analysis (PCA) and its probabilistic extension, Mixtures of Probabilistic PCA (MPPCA), are fundamental tools for dimensionality reduction and clustering. MPPCA models data as a mixture of several low-dimensional subspaces, effectively performing clustering and low-rank representation simultaneously. However, a significant limitation of MPPCA is its assumption of homoscedastic noise—the premise that every data point within a cluster is corrupted by an identical noise variance.

In modern data science, datasets are often aggregated from diverse sources, such as different sensors or varying environments, leading to heteroscedastic noise (varying noise levels across samples). Standard MPPCA fails to distinguish between the underlying signal structure and these varying noise levels, leading to suboptimal performance. This paper introduces HeMPPCAT (Heteroscedastic MPPCA Technique) to address this challenge by explicitly modeling sample-specific noise variances.

Methodology: The HeMPPCAT Framework

HeMPPCAT extends the MPPCA model by decoupling noise characteristics from the mixture components. The proposed model assumes each sample belongs to one of $J$ mixture components (clusters) and one of $L$ noise groups. This flexibility allows the model to handle scenarios where noise is independent of the data's category.

To estimate the model parameters, the authors developed a Generalized Expectation-Maximization (GEM) algorithm. This iterative approach jointly optimizes mixing proportions, noise variances for each of the $L$ groups, and the means and factor matrices defining the underlying subspaces. By estimating noise variance per group, the GEM algorithm naturally "down-weights" the influence of highly noisy samples, providing a built-in robustness mechanism that prevents outliers from biasing the estimated subspaces.

Experimental Validation

The performance of HeMPPCAT was evaluated using both synthetic and real-world datasets.

Synthetic Experiments

The researchers generated synthetic datasets with varying levels of noise heterogeneity to assess how accurately the algorithms could recover ground-truth factor matrices. As shown in the following plots, as the noise heterogeneity (represented by $v_1$) increases, the estimation error for standard MPPCA and K-Planes rises sharply. In contrast, HeMPPCAT maintains significantly lower and more stable error rates across all factors.

Factor estimation errors for F1, F2, and F3 across varying noise levels.

Real-World Application: Motion Segmentation

HeMPPCAT was also tested on the Hopkins 155 dataset, a benchmark for motion segmentation. To simulate real-world "non-rigid" motion, synthetic noise was added to the video sequences. The results, summarized in the table below, show that HeMPPCAT consistently outperforms both MPPCA and the non-statistical K-Planes method across all noise levels. The improvement is most pronounced in the "high noise" group, where HeMPPCAT's misclassification rate is nearly 15 percentage points lower than that of MPPCA.

Misclassification rates on the Hopkins 155 dataset across different noise levels.

Discussion and Future Work

HeMPPCAT represents a significant advancement in handling multi-source data. By allowing noise groups to exist independently of clusters, the model gains the necessary flexibility for real-world applications. Future research could focus on automated parameter selection for $J$ and $L$, as well as exploring feature-wise noise variance. Additionally, computational optimizations like Space-Alternating Generalized EM (SAGE) could be employed to improve convergence speeds for massive datasets.

Summary updated

Abstract

Mixtures of probabilistic principal component analysis (MPPCA) is a well-known mixture model extension of principal component analysis (PCA). Similar to PCA, MPPCA assumes the data samples in each mixture contain homoscedastic noise. However, datasets with heterogeneous noise across samples are becoming increasingly common, as larger datasets are generated by collecting samples from several sources with varying noise profiles. The performance of MPPCA is suboptimal for data with heteroscedastic noise across samples. This paper proposes a heteroscedastic mixtures of probabilistic PCA technique (HeMPPCAT) that uses a generalized expectation-maximization (GEM) algorithm to jointly estimate the unknown underlying factors, means, and noise variances under a heteroscedastic noise setting. Simulation results illustrate the improved factor estimates and clustering accuracies of HeMPPCAT compared to MPPCA.
BibTeX
@inproceedings{xu2023hemppcat,
  author = {Alec S. Xu and Laura Balzano and Jeffrey A. Fessler},
  title = {{HeMPPCAT: Mixtures of Probabilistic Principal Component Analysers for Data with Heteroscedastic Noise}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=4hNv7oFqBk},
}
Figure