Exponential Convergence of CAVI for Bayesian PCA

Arghya Datta, Philippe Gagnon, Florian MairearXiv 2025 · DOI · arXiv

Introduction

Principal Component Analysis (PCA) is a cornerstone of dimensionality reduction, yet its classical formulation lacks a probabilistic framework, making it difficult to quantify uncertainty in its estimates. Bayesian Principal Component Analysis (BPCA) addresses this by treating principal components as latent variables within a generative model. To perform inference in BPCA, researchers typically rely on Coordinate Ascent Variational Inference (CAVI), an iterative algorithm that updates groups of variables sequentially. While CAVI is widely used due to its implementation simplicity, its theoretical convergence properties for BPCA have remained largely uncharacterized until now.

This paper provides the first rigorous proof that CAVI converges exponentially fast when applied to BPCA. By bridging Bayesian statistics with classical numerical linear algebra, the authors demonstrate that BPCA is not merely a heuristic for uncertainty quantification but a theoretically sound and computationally efficient numerical scheme.

Theoretical Framework

The authors establish convergence through two primary analytical paths:

  1. The Single Component Case ($k=1$): The study reveals a mathematical equivalence between CAVI for a single-component BPCA and the Power Iteration algorithm, a staple of numerical linear algebra used to find the dominant eigenvalue of a matrix. This connection allows the application of classical numerical analysis results to prove global exponential convergence.
  2. The General Multi-Component Case ($k > 1$): For multiple components, the authors utilize a modern framework for variational inference convergence. A key technical contribution here is the development of a novel lower bound for the symmetric Kullback–Leibler (KL) divergence between multivariate normal distributions. This bound is sharper than traditional tools like Pinsker’s inequality and ensures the "local contraction" of the algorithm toward a stationary point.

Numerical Evidence

To validate the theoretical findings, the authors conducted experiments using simulated datasets where ground truth parameters were known (e.g., $n=100, d=10$). The results consistently show that the error decreases linearly on a log scale, which is the defining characteristic of exponential convergence.

Exponential convergence of latent variable iterates compared to theoretical upper bounds.

As seen in the figure above, the iterates for the latent scores ($\mu_Z$) and loading matrices ($\mu_W$) track closely with the predicted theoretical upper bounds, often reaching optimal scaling values within very few iterations. Further long-term simulations confirm that this rate is sustained over thousands of iterations for various model parameters.

Log-error of model parameters over 8000 iterations, demonstrating sustained linear convergence on a logarithmic scale.

Key Insights

  • The Power Iteration Link: The realization that Bayesian inference performs a "noisy" version of a classical linear algebra routine provides a deep intuition for why CAVI is so effective for PCA-like problems.
  • The Role of the Prior: The research highlights that the choice of prior is critical. If the prior is too uninformative (proportional to the identity matrix), the model suffers from rotational ambiguity, leading to a flat loss surface and optimization difficulties. A "non-proportional" prior is necessary to ensure a unique, fast-converging solution.
  • Convergence Rates: For $k=1$, the convergence rate is governed by the spectral gap of the data, specifically the ratio $(\lambda_1 - \lambda_2)/\lambda_1$. A dominant first principal component leads to extremely rapid convergence.

Conclusion and Future Work

While this work provides a robust foundation for understanding CAVI in the context of BPCA, some limitations remain. The proof for the multi-component case is currently a local result, guaranteeing convergence only when the algorithm is initialized reasonably well. Additionally, the study assumes fixed hyperparameters (such as the number of components $k$), whereas practitioners often learn these from data. Future research into global convergence for $k > 1$ and the behavior of the algorithm with learned hyperparameters will further solidify the theoretical standing of Bayesian dimensionality reduction.

Summary updated

Abstract

Probabilistic principal component analysis (PCA) and its Bayesian variant (BPCA) are widely used for dimension reduction in machine learning and statistics. The main advantage of probabilistic PCA over the traditional formulation is allowing uncertainty quantification. The parameters of BPCA are typically learned using mean-field variational inference, and in particular, the coordinate ascent variational inference (CAVI) algorithm. So far, the convergence speed of CAVI for BPCA has not been characterized. In our paper, we fill this gap in the literature. Firstly, we prove a precise exponential convergence result in the case where the model uses a single principal component (PC). Interestingly, this result is established through a connection with the classical $\textit{power iteration algorithm}$ and it indicates that traditional PCA is retrieved as points estimates of the BPCA parameters. Secondly, we leverage recent tools to prove exponential convergence of CAVI for the model with any number of PCs, thus leading to a more general result, but one that is of a slightly different flavor. To prove the latter result, we additionally needed to introduce a novel lower bound for the symmetric Kullback--Leibler divergence between two multivariate normal distributions, which, we believe, is of independent interest in information theory.
BibTeX
@misc{datta2025exponential,
  author = {Arghya Datta and Philippe Gagnon and Florian Maire},
  title = {{Exponential Convergence of CAVI for Bayesian PCA}},
  year = {2025},
  eprint = {2505.16145},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.16145},
}
Figure