Managing sparsity, time, and quality of inference in topic models

Khoat Than, Tu Bao HoarXiv 2012 · OpenReview

Abstract

Inference is an integral part of probabilistic topic models, but is often non-trivial to derive an efficient algorithm for a specific model. It is even much more challenging when we want to find a fast inference algorithm which always yields sparse latent representations of documents. In this article, we introduce a simple framework for inference in probabilistic topic models, denoted by FW. This framework is general and flexible enough to be easily adapted to mixture models. It has a linear convergence rate, offers an easy way to incorporate prior knowledge, and provides us an easy way to directly trade off sparsity against quality and time. We demonstrate the goodness and flexibility of FW over existing inference methods by a number of tasks. Finally, we show how inference in topic models with nonconjugate priors can be done efficiently.

Introduction

Probabilistic topic modeling, such as Latent Dirichlet Allocation (LDA), is essential for discovering latent themes in large text collections. However, the inference process—determining the topic distribution for new documents—presents a persistent "trilemma": balancing computational speed, document sparsity, and predictive quality. Standard methods like Variational Bayes (VB) often produce dense representations that are computationally expensive and difficult to interpret. Furthermore, complex nonconjugate models like Correlated Topic Models (CTM) have long been considered computationally difficult.

The FW Framework

The proposed FW framework addresses these challenges by framing inference as a constrained concave maximization problem over a simplex. It utilizes the Frank-Wolfe (Conditional Gradient) algorithm, a greedy iterative approach.

Key Features:

  • Sparsity by Design: A fundamental property of the Frank-Wolfe algorithm is that after $k$ iterations, the solution contains at most $k+1$ non-zero components. This allows users to treat sparsity as a controllable "budget," stopping the algorithm early for high sparsity or running it longer for higher precision.
  • Generality: The framework is model-agnostic, requiring only a continuously differentiable, concave objective function (e.g., log-likelihood).
  • Theoretical Tractability: The authors prove that inference in nonconjugate models like CTM is a concave optimization problem with a linear convergence rate, making these complex models viable for large-scale use.

Experimental Evaluation

The framework was tested on seven diverse corpora, including AP, NIPS, and Enron. The evaluation focused on speed, sparsity, and predictive power (perplexity).

Results and Insights

  • Performance: FW significantly outperformed Variational Bayes (VB) and standard "Folding-in" heuristics in terms of speed, particularly as the number of topics ($K$) increased.
  • Sparsity vs. Quality: While VB typically yields 100% dense representations, FW maintains high sparsity without sacrificing predictive quality. In fact, FW often achieved lower perplexity scores than VB.
  • Clustering: Sparse representations generated by FW led to better document separability in the topical space, forming clearer clusters compared to the "smear" often produced by dense methods.
  • Convergence: The algorithm typically reaches high-quality solutions within 20–30 iterations, regardless of the total topic count.

Conclusion

The FW framework provides a flexible and efficient solution for inference in both conjugate and nonconjugate topic models. By allowing direct control over the sparsity-quality-time trade-off, it enables more scalable and interpretable text analysis. Future research may focus on optimizing objective functions for supervised learning and refining boundary handling for specific nonconjugate priors.

Summary updated

Abstract

Inference is an integral part of probabilistic topic models, but is often non-trivial to derive an efficient algorithm for a specific model. It is even much more challenging when we want to find a fast inference algorithm which always yields sparse latent representations of documents. In this article, we introduce a simple framework for inference in probabilistic topic models, denoted by FW. This framework is general and flexible enough to be easily adapted to mixture models. It has a linear convergence rate, offers an easy way to incorporate prior knowledge, and provides us an easy way to directly trade off sparsity against quality and time. We demonstrate the goodness and flexibility of FW over existing inference methods by a number of tasks. Finally, we show how inference in topic models with nonconjugate priors can be done efficiently.
BibTeX
@misc{than2012managing,
  author = {Khoat Than and Tu Bao Ho},
  title = {{Managing sparsity, time, and quality of inference in topic models}},
  year = {2012},
  eprint = {1210.7053},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1210.7053},
}
Figure