TL;DR
This paper demonstrates that the choice of Dirichlet priors in Latent Dirichlet Allocation (LDA) isn't just a minor technical detail—it’s a fundamental modeling choice. By using an asymmetric prior for document-topic distributions and a symmetric prior for topic-word distributions, researchers can make LDA significantly more robust to stop words and less sensitive to the chosen number of topics ($T$).
What Problem Does This Paper Address?
Latent Dirichlet Allocation (LDA) is the "bread and butter" of topic modeling, but practitioners often struggle with two major headaches:
- The Stop Word Problem: Common words (like "the," "and," "of") often dominate every topic, forcing researchers to manually curate extensive "stop word" lists.
- The $T$ Problem: Choosing the number of topics ($T$) feels like guesswork. If $T$ is set too high, the model often shatters meaningful topics into uninterpretable fragments.
Most researchers traditionally used "symmetric" Dirichlet priors (assuming every topic is equally likely a priori) with fixed parameters. This paper investigates whether the structure of these priors—specifically moving to asymmetric distributions—can solve these practical implementation hurdles.
Methods: Exploring Prior Structures
The authors systematically compare four combinations of priors:
- SS: Symmetric document-topic, Symmetric topic-word (The traditional standard).
- SA: Symmetric document-topic, Asymmetric topic-word.
- AS: Asymmetric document-topic, Symmetric topic-word.
- AA: Asymmetric document-topic, Asymmetric topic-word.
They utilize a Hierarchical Bayesian approach, where the base measures of the Dirichlet distributions are themselves treated as variables to be inferred. Because full Bayesian inference is computationally expensive, they also introduce ASO (Asymmetric-Symmetric Optimized), which uses efficient hyperparameter optimization steps within the standard Gibbs sampling inference. This provides the performance gains of the full Bayesian model at a fraction of the time.
Datasets
The study evaluates these models on three diverse text collections:
- Patent Abstracts: Specifically those related to carbon nanotechnology (1,016 documents).
- 20 Newsgroups: A classic benchmark of Usenet postings (540 documents used in the subset).
- New York Times Articles: A larger corpus of news stories (1,768 documents used in the subset).
Key Insights and Ideas
- The "AS" Sweet Spot: The authors found that the AS configuration (Asymmetric document-topic, Symmetric topic-word) is the clear winner.
- Why Asymmetry for Documents? In any real-world corpus, some topics are just more common than others. An asymmetric prior allows the model to "sequester" common words and stop words into a few high-frequency topics, leaving the remaining topics "clean" and semantically meaningful.
- Why Symmetry for Words? Topics are supposed to be distinct. If you use an asymmetric prior for topic-word distributions, the model tries to make every topic look like the global word frequency (i.e., every topic starts looking like a list of stop words).
- "Nibbling" vs. Splitting: In symmetric models, increasing $T$ causes existing topics to split apart. In the AS model, the core topics remain stable; new topics simply "nibble away" small, specific clusters of words without destroying the overall structure.
Key Results
- Better Likelihood: The AS and AA models consistently achieved higher log-likelihood on held-out data compared to the standard SS model.
- Efficiency: The optimized version (ASO) performed nearly as well as the full hierarchical Bayesian model but was over 8x faster (30 minutes vs. 4 hours for the patent dataset).
- Robustness to $T$: Using Variation of Information (VI) as a metric, the authors showed that the AS model's topic assignments remained much more consistent as $T$ increased from 25 to 100, whereas the standard model's assignments changed drastically.
- Stop Word Resilience: The AS model produced high-quality topics even when stop words were not removed, effectively isolating them into their own "junk" topics.
Limitations and Future Work
- Complexity of AA: While the fully asymmetric (AA) model can match the performance of AS, it has many more degrees of freedom, making it inconsistent and prone to getting stuck in poor local optima.
- Finite vs. Infinite: The paper focuses on finite LDA. While non-parametric models (like HDP) exist to handle unknown $T$, the authors argue that an optimized AS model provides the benefits of data-driven topic counts while maintaining the simplicity and speed of finite models.
- Standardization: The primary takeaway is a call to action: the authors recommend that asymmetric document-topic priors should become the new default standard for all LDA implementations.
Summary updated