Rethinking LDA: Why Priors Matter

Hanna M. Wallach, David M. Mimno, Andrew McCallumNeurIPS 2009 · OpenReview

TL;DR

This paper demonstrates that the choice of Dirichlet priors in Latent Dirichlet Allocation (LDA) isn't just a minor technical detail—it’s a fundamental modeling choice. By using an asymmetric prior for document-topic distributions and a symmetric prior for topic-word distributions, researchers can make LDA significantly more robust to stop words and less sensitive to the chosen number of topics ($T$).

What Problem Does This Paper Address?

Latent Dirichlet Allocation (LDA) is the "bread and butter" of topic modeling, but practitioners often struggle with two major headaches:

  1. The Stop Word Problem: Common words (like "the," "and," "of") often dominate every topic, forcing researchers to manually curate extensive "stop word" lists.
  2. The $T$ Problem: Choosing the number of topics ($T$) feels like guesswork. If $T$ is set too high, the model often shatters meaningful topics into uninterpretable fragments.

Most researchers traditionally used "symmetric" Dirichlet priors (assuming every topic is equally likely a priori) with fixed parameters. This paper investigates whether the structure of these priors—specifically moving to asymmetric distributions—can solve these practical implementation hurdles.

Methods: Exploring Prior Structures

The authors systematically compare four combinations of priors:

  • SS: Symmetric document-topic, Symmetric topic-word (The traditional standard).
  • SA: Symmetric document-topic, Asymmetric topic-word.
  • AS: Asymmetric document-topic, Symmetric topic-word.
  • AA: Asymmetric document-topic, Asymmetric topic-word.
Graphical models of LDA variants

They utilize a Hierarchical Bayesian approach, where the base measures of the Dirichlet distributions are themselves treated as variables to be inferred. Because full Bayesian inference is computationally expensive, they also introduce ASO (Asymmetric-Symmetric Optimized), which uses efficient hyperparameter optimization steps within the standard Gibbs sampling inference. This provides the performance gains of the full Bayesian model at a fraction of the time.

Datasets

The study evaluates these models on three diverse text collections:

  • Patent Abstracts: Specifically those related to carbon nanotechnology (1,016 documents).
  • 20 Newsgroups: A classic benchmark of Usenet postings (540 documents used in the subset).
  • New York Times Articles: A larger corpus of news stories (1,768 documents used in the subset).

Key Insights and Ideas

  • The "AS" Sweet Spot: The authors found that the AS configuration (Asymmetric document-topic, Symmetric topic-word) is the clear winner.
  • Why Asymmetry for Documents? In any real-world corpus, some topics are just more common than others. An asymmetric prior allows the model to "sequester" common words and stop words into a few high-frequency topics, leaving the remaining topics "clean" and semantically meaningful.
  • Why Symmetry for Words? Topics are supposed to be distinct. If you use an asymmetric prior for topic-word distributions, the model tries to make every topic look like the global word frequency (i.e., every topic starts looking like a list of stop words).
  • "Nibbling" vs. Splitting: In symmetric models, increasing $T$ causes existing topics to split apart. In the AS model, the core topics remain stable; new topics simply "nibble away" small, specific clusters of words without destroying the overall structure.

Key Results

  • Better Likelihood: The AS and AA models consistently achieved higher log-likelihood on held-out data compared to the standard SS model.
  • Efficiency: The optimized version (ASO) performed nearly as well as the full hierarchical Bayesian model but was over 8x faster (30 minutes vs. 4 hours for the patent dataset).
  • Robustness to $T$: Using Variation of Information (VI) as a metric, the authors showed that the AS model's topic assignments remained much more consistent as $T$ increased from 25 to 100, whereas the standard model's assignments changed drastically.
  • Stop Word Resilience: The AS model produced high-quality topics even when stop words were not removed, effectively isolating them into their own "junk" topics.

Limitations and Future Work

  • Complexity of AA: While the fully asymmetric (AA) model can match the performance of AS, it has many more degrees of freedom, making it inconsistent and prone to getting stuck in poor local optima.
  • Finite vs. Infinite: The paper focuses on finite LDA. While non-parametric models (like HDP) exist to handle unknown $T$, the authors argue that an optimized AS model provides the benefits of data-driven topic counts while maintaining the simplicity and speed of finite models.
  • Standardization: The primary takeaway is a call to action: the authors recommend that asymmetric document-topic priors should become the new default standard for all LDA implementations.

Summary updated

Abstract

Implementations of topic models typically use symmetric Dirichlet priors with fixed concentration parameters, with the implicit assumption that such smoothing parameters" have little practical effect. In this paper, we explore several classes of structured priors for topic models. We find that an asymmetric Dirichlet prior over the document-topic distributions has substantial advantages over a symmetric prior, while an asymmetric prior over the topic-word distributions provides no real benefit. Approximation of this prior structure through simple, efficient hyperparameter optimization steps is sufficient to achieve these performance gains. The prior structure we advocate substantially increases the robustness of topic models to variations in the number of topics and to the highly skewed word frequency distributions common in natural language. Since this prior structure can be implemented using efficient algorithms that add negligible cost beyond standard inference techniques, we recommend it as a new standard for topic modeling."
BibTeX
@inproceedings{wallach2009rethinking,
  author = {Hanna M. Wallach and David M. Mimno and Andrew McCallum},
  title = {{Rethinking LDA: Why Priors Matter}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2009},
  url = {https://openreview.net/forum?id=ByEzxFZdbB},
}
Figure