Transformations for variational factor analysis to speed up learning

Jaakko Luttinen, Alexander Ilin, Tapani RaikoESANN 2009

Abstract

Variational Bayesian (VB) methods are widely used for latent variable models but are often hampered by slow convergence rates. This paper presents a method to accelerate VB learning in factor analysis by applying centering and whitening transformations to the latent space. By decorrelating the hidden states and loading matrices, these transformations mitigate the "coupling problem" that typically slows down coordinate-ascent style updates. Experimental results on artificial datasets demonstrate that this approach achieves convergence orders of magnitude faster than standard methods while improving the model's ability to identify the true underlying dimensionality of the data.

Introduction

Factor analysis models, including Principal Component Analysis (PCA), assume that high-dimensional observed data is generated by a lower-dimensional set of hidden states ($x$) transformed by a loading matrix ($W$). In the Variational Bayesian framework, learning involves approximating the posterior distributions of these latent variables. This is usually achieved through an iterative process where the distribution of $x$ is updated while $W$ is held constant, and vice versa.

However, this "EM-like" update scheme suffers from a fundamental efficiency issue known as the coupling problem. Because the product $W \cdot x$ is invariant to certain transformations (e.g., if $W$ is multiplied by a scalar and $x$ is divided by the same scalar), the variables become interdependent. During optimization, an update to $x$ is restricted by the current, potentially suboptimal, state of $W$. This results in a slow, incremental progression toward the global optimum, often characterized by a "zig-zag" trajectory in the parameter space.

Methodology: Optimizing the Variational Lower Bound

The authors propose a mathematically rigorous solution: applying transformations to the hidden states that are derived directly from the optimization of the Variational Lower Bound (ELBO), also known as the negative VB cost.

  • Centering (Translation): The authors show that the "bias" or mean can be shifted between the hidden states and the global mean parameter $\mu$. By minimizing the VB cost with respect to this shift, they find that the latent variables should always be transformed to have a mean of zero.
  • Whitening (Rotation and Scaling): The loading matrix $W$ can be rotated arbitrarily if the inverse rotation is applied to $X$. Using Singular Value Decomposition (SVD), the authors derive a rotation matrix $R$ that whitens the hidden states—making their covariance an identity matrix—while simultaneously orthogonalizing the columns of $W$.

Crucially, these transformations are not just pre-processing heuristics. They are active adjustments to the approximate posterior distributions. By aligning the posterior more closely with the prior, these steps reduce the Kullback-Leibler (KL) divergence term in the ELBO without negatively impacting the data likelihood. Since these operations occur in the low-dimensional latent space, the computational overhead is negligible compared to the resulting gains in convergence speed.

Experimental Evaluation

The effectiveness of the proposed transformations was tested on an artificial dataset with $N=500$ samples and $M=50$ dimensions. The true data was generated from a 10-dimensional subspace. To evaluate the model's predictive capabilities and robustness to missing data, 20% of the observations were withheld during training.

Comparison of convergence rates for standard VB and VB with transformations. Subfigure (a) shows VB cost, (b) Training RMSE, and (c) Validation RMSE, all plotted against time on a logarithmic scale.

As illustrated in Figure 4, the inclusion of transformations leads to a dramatic increase in learning speed. On the logarithmic time axis, the transformed model (dotted line) reaches a stable minimum for the VB cost and RMSE significantly earlier than the standard model (solid line).

Beyond raw speed, the transformations also improved the model's structural learning. In VB PCA, the algorithm is designed to automatically prune unnecessary dimensions. Without transformations, the model struggled to distinguish signal from noise, leading to prolonged periods of overfitting. In contrast, the transformed model identified the correct 10-dimensional structure almost immediately, demonstrating superior stability and accuracy in dimensionality detection.

Discussion and Conclusion

The "invariance insight" presented in this paper—exploiting the coordinate-system "slack" in latent variable models—provides a powerful tool for speeding up Variational Bayesian inference. By centering and whitening the latent space, the authors effectively decorrelate the parameters, allowing for much larger and more effective steps during optimization.

While the explicit derivations in this work are tailored for linear factor analysis, the underlying principle is broadly applicable. The authors suggest that similar transformations could be derived for a wide range of models, including those in the exponential family or those utilizing nonlinear mappings. This research provides a theoretical and practical blueprint for making Variational Bayesian methods more viable for large-scale, high-dimensional data analysis.

Summary updated

BibTeX
@inproceedings{luttinen2009transformations,
  author = {Jaakko Luttinen and Alexander Ilin and Tapani Raiko},
  title = {{Transformations for variational factor analysis to speed up learning}},
  booktitle = {ESANN},
  year = {2009},
}
Figure