On the Consistency of Maximum Likelihood Estimation of Probabilistic Principal Component Analysis

Arghya Datta, Sayak Chakrabarty

Probabilistic Principal Component Analysis (PPCA) is a fundamental tool in modern data science, extending standard PCA into a statistical framework that facilitates uncertainty quantification and the handling of missing data. Despite its widespread adoption since its introduction by Tipping and Bishop in 1999, a rigorous proof of its "consistency"—the mathematical guarantee that the model converges to the true parameters as the sample size increases—has remained elusive. This paper addresses this theoretical gap by reframing the PPCA parameter space using quotient topological spaces, effectively resolving the rotational ambiguity that previously hindered formal proofs.

The Challenge of Identifiability

In standard PCA, the goal is to identify directions of maximum variance. PPCA formalizes this as a latent variable model where observed data is generated from a lower-dimensional latent space via a loading matrix $W$. However, PPCA suffers from a significant theoretical hurdle: identifiability.

Specifically, if a loading matrix $W$ describes the data, any rotated version $WR$ (where $R$ is an orthogonal rotation matrix) yields the same covariance structure and is thus mathematically indistinguishable to the model. Because there is no unique "correct" matrix, but rather an infinite set of equivalent matrices, classical statistical proofs—such as Wald’s consistency theorems—cannot be directly applied. These theorems typically require a unique parameter to converge toward, and the "drift" within the set of equivalent rotations prevented a definitive proof of Maximum Likelihood Estimate (MLE) consistency as the sample size grew to infinity.

Methodology: Navigating Quotient Spaces

The authors overcome the identifiability problem by redefining the parameter space. Instead of attempting to isolate a single matrix $W$, they treat the entire set of rotated matrices as a single point within a quotient space. This approach effectively "maths away" the ambiguity by grouping all parameters that produce the same covariance matrix into equivalence classes.

Key technical steps in this proof include:

  1. Defining the Quotient Space: Parameters are grouped into "equivalence classes" based on the covariance they generate. All rotated versions of a matrix are treated as one entity.
  2. Establishing Metrizability: The authors define a specific metric to measure distances within this abstract quotient space. This is a non-trivial task, as the relationship between the "natural" distance of classes and the underlying topology must be carefully managed to ensure the space behaves predictably.
  3. Extending Wald’s Framework: By adapting Wald’s classic consistency conditions to operate on these equivalence classes rather than individual points, the authors prove that the MLE "bucket" of answers converges to the "true bucket."
  4. Handling Dependent Data: The proof is extended beyond independent and identically distributed (i.i.d.) samples. Using Bernstein’s weak law of large numbers, the authors demonstrate that consistency holds even for data with "local dependence," such as time series.

Key Theoretical Insights

The core innovation of this work is the "Bucket" approach. By shifting the focus from finding a specific rotation to identifying the correct equivalence class, the authors demonstrate that the inherent ambiguity of PPCA does not preclude statistical certainty.

A subtle but vital distinction made in the paper is the difference between Metric and Topology. The authors highlight that in abstract spaces, the way distance is measured (metric) and the way closeness is defined (topology) can diverge. They meticulously prove that their chosen metric is compatible with the quotient topology, providing the necessary foundation for the statistical proofs. Furthermore, the results are not limited to the MLE; they apply to a broader class of estimators that satisfy specific likelihood conditions.

Results and Implications

The paper establishes two primary forms of consistency:

  • Strong Consistency: Under the assumption of a compact parameter space, the MLE for PPCA converges "almost surely" to the true parameter class in the quotient space.
  • Weak Consistency: In scenarios involving dependent data, the estimate still converges in probability, provided the data satisfies a weak law of large numbers.

These results provide the first rigorous justification for the MLE solutions used for over two decades. For practitioners, this confirms that the estimated covariance in a PPCA model is strongly consistent, solidifying its use in density estimation, noise reduction, and uncertainty quantification.

Limitations and Future Directions

While the paper proves that the estimate will eventually reach the right answer, it does not address the rate of convergence. The authors identify two primary areas for future research:

  1. Rate of Convergence: Establishing the speed (e.g., $1/\sqrt{n}$) at which the estimate approaches the true equivalence class.
  2. Asymptotic Distribution: Determining the "shape" of the error distribution in the quotient space, which is essential for developing formal hypothesis tests and confidence intervals.

The framework developed here—using quotient spaces to handle unidentifiable models—holds potential for other complex problems in statistics, such as matrix factorization ($X = AB + \text{noise}$) and non-linear independent component analysis.

Summary updated

Abstract

Probabilistic principal component analysis (PPCA) is currently one of the most used statistical tools to reduce the ambient dimension of the data. From multidimensional scaling to the imputation of missing data, PPCA has a broad spectrum of applications ranging from science and engineering to quantitative finance.\\ Despite this wide applicability in various fields, hardly any theoretical guarantees exist to justify the soundness of the maximal likelihood (ML) solution for this model. In fact, it is well known that the maximum likelihood estimation (MLE) can only recover the true model parameters up to a rotation. The main obstruction is posed by the inherent identifiability nature of the PPCA model resulting from the rotational symmetry of the parameterization. To resolve this ambiguity, we propose a novel approach using quotient topological spaces and in particular, we show that the maximum likelihood solution is consistent in an appropriate quotient Euclidean space. Furthermore, our consistency results encompass a more general class of estimators beyond the MLE. Strong consistency of the ML estimate and consequently strong covariance estimation of the PPCA model have also been established under a compactness assumption.
BibTeX
@inproceedings{datta2023consistency,
  author = {Arghya Datta and Sayak Chakrabarty},
  title = {{On the Consistency of Maximum Likelihood Estimation of Probabilistic Principal Component Analysis}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=q0RfX96un8},
}
Figure