Latent Gaussian Models for Topic Modeling

Changwei Hu, Eunsu Ryu, David E. Carlson, Yingjian Wang, Lawrence CarinAISTATS 2014 · OpenReview

Abstract

A new approach is proposed for topic modeling, in which the latent matrix factorization employs Gaussian priors, rather than the Dirichlet-class priors widely used in such models. The use of a Latent Gaussian Model (LGM) allows for efficient inference via the Laplace approximation, transforming the learning process into a high-performance optimization problem. This framework not only achieves state-of-the-art accuracy but also significantly reduces computational costs and facilitates the joint modeling of text and auxiliary binary data.

Introduction

Topic modeling is a cornerstone of text analysis, enabling the discovery of latent thematic structures within large document corpora. Traditionally, models like Latent Dirichlet Allocation (LDA) have relied on Dirichlet or Gamma distributions to ensure that word and topic proportions are non-negative and sum to one. While theoretically sound, these priors impose a "computational tax," necessitating complex Markov Chain Monte Carlo (MCMC) sampling or Variational Bayes (VB) methods. These approaches often struggle with slow convergence and limited parallelization, posing challenges for scaling to millions of documents or integrating multi-modal data.

Methodology: Latent Gaussian Models

The proposed Latent Gaussian Models (LGM) shift the paradigm by placing Gaussian priors on the latent matrices—specifically, the topic matrix $D$ and the document loading matrix $S$. This choice enables the application of the Laplace approximation for Bayesian posterior inference, effectively recasting the inference task as an optimization problem.

LG-NMF and Laplace Approximation

A key variant, LG-NMF (Latent Gaussian Non-negative Matrix Factorization), enforces non-negativity constraints on the factors, ensuring compatibility with standard topic modeling interpretations. By leveraging ideas from Integrated Nested Laplace Approximation (INLA), the authors handle hyperparameters by gridding a small hyperparameter space. This allows for the underlying optimization problems to be solved efficiently and in parallel.

Visualization of the posterior surface over hyperparameters, illustrating the effectiveness of the gridding approach.

Experimental Results

The researchers evaluated LG-NMF against standard benchmarks, including the PsyRev and NIPS datasets. The results demonstrate that Gaussian priors, despite being computationally cheaper, yield perplexity scores comparable to or better than traditional LDA and more complex models like the Focused Topic Model (FTM).

Efficiency and Convergence

One of the most significant advantages of the LGM framework is its speed. LG-NMF converges substantially faster than MCMC-based alternatives. As shown in Figure 1, the model reaches a stable, low perplexity state in a fraction of the time required by competing methods.

Perplexity vs. Time (log10 seconds) for various models. LG-NMF (red) demonstrates rapid convergence compared to LDA and FTM.

Scalability and Joint Modeling

The LGM framework naturally extends to multi-modal data. In a joint analysis of legislative text and voting records from the US House of Representatives, the model achieved a 90.65% prediction accuracy, matching state-of-the-art performance with significantly lower computational overhead.

For massive datasets, an "online" version using Stochastic Gradient Descent (SGD) was implemented. When tested on a corpus of 1 million Wikipedia documents, the parallelized LG-NMF completed processing in approximately 8 hours, a stark contrast to the 90 hours required by standard online LDA-VB implementations.

Conclusion

By replacing Dirichlet-class priors with Latent Gaussian priors, this work bridges the gap between Bayesian inference and high-performance optimization. The resulting models offer a compelling combination of accuracy, speed, and scalability. Future work may involve exploring more complex "nested" Laplace approximations to further refine precision and integrating advanced optimization algorithms to enhance robustness.

Summary updated

Abstract

A new approach is proposed for topic modeling, in which the latent matrix factorization employs Gaussian priors, rather than the Dirichlet-class priors widely used in such models. The use of a late...
BibTeX
@inproceedings{hu2014latent,
  author = {Changwei Hu and Eunsu Ryu and David E. Carlson and Yingjian Wang and Lawrence Carin},
  title = {{Latent Gaussian Models for Topic Modeling}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2014},
  url = {https://openreview.net/forum?id=Jqfhzuaz_VK},
}

Loading related content...

Figure