Abstract
A new approach is proposed for topic modeling, in which the latent matrix factorization employs Gaussian priors, rather than the Dirichlet-class priors widely used in such models. The use of a Latent Gaussian Model (LGM) allows for efficient inference via the Laplace approximation, transforming the learning process into a high-performance optimization problem. This framework not only achieves state-of-the-art accuracy but also significantly reduces computational costs and facilitates the joint modeling of text and auxiliary binary data.
Introduction
Topic modeling is a cornerstone of text analysis, enabling the discovery of latent thematic structures within large document corpora. Traditionally, models like Latent Dirichlet Allocation (LDA) have relied on Dirichlet or Gamma distributions to ensure that word and topic proportions are non-negative and sum to one. While theoretically sound, these priors impose a "computational tax," necessitating complex Markov Chain Monte Carlo (MCMC) sampling or Variational Bayes (VB) methods. These approaches often struggle with slow convergence and limited parallelization, posing challenges for scaling to millions of documents or integrating multi-modal data.
Methodology: Latent Gaussian Models
The proposed Latent Gaussian Models (LGM) shift the paradigm by placing Gaussian priors on the latent matrices—specifically, the topic matrix $D$ and the document loading matrix $S$. This choice enables the application of the Laplace approximation for Bayesian posterior inference, effectively recasting the inference task as an optimization problem.
LG-NMF and Laplace Approximation
A key variant, LG-NMF (Latent Gaussian Non-negative Matrix Factorization), enforces non-negativity constraints on the factors, ensuring compatibility with standard topic modeling interpretations. By leveraging ideas from Integrated Nested Laplace Approximation (INLA), the authors handle hyperparameters by gridding a small hyperparameter space. This allows for the underlying optimization problems to be solved efficiently and in parallel.
Experimental Results
The researchers evaluated LG-NMF against standard benchmarks, including the PsyRev and NIPS datasets. The results demonstrate that Gaussian priors, despite being computationally cheaper, yield perplexity scores comparable to or better than traditional LDA and more complex models like the Focused Topic Model (FTM).
Efficiency and Convergence
One of the most significant advantages of the LGM framework is its speed. LG-NMF converges substantially faster than MCMC-based alternatives. As shown in Figure 1, the model reaches a stable, low perplexity state in a fraction of the time required by competing methods.
Scalability and Joint Modeling
The LGM framework naturally extends to multi-modal data. In a joint analysis of legislative text and voting records from the US House of Representatives, the model achieved a 90.65% prediction accuracy, matching state-of-the-art performance with significantly lower computational overhead.
For massive datasets, an "online" version using Stochastic Gradient Descent (SGD) was implemented. When tested on a corpus of 1 million Wikipedia documents, the parallelized LG-NMF completed processing in approximately 8 hours, a stark contrast to the 90 hours required by standard online LDA-VB implementations.
Conclusion
By replacing Dirichlet-class priors with Latent Gaussian priors, this work bridges the gap between Bayesian inference and high-performance optimization. The resulting models offer a compelling combination of accuracy, speed, and scalability. Future work may involve exploring more complex "nested" Laplace approximations to further refine precision and integrating advanced optimization algorithms to enhance robustness.
Summary updated