Decoupling Sparsity and Smoothness in the Discrete Hierarchical Dirichlet Process

Chong Wang, David M. BleiNeurIPS 2009 · OpenReview

TL;DR

The Sparse Topic Model (sparseTM) improves upon the traditional Hierarchical Dirichlet Process (HDP) by decoupling two previously linked properties: sparsity (how many unique words a topic uses) and smoothness (how evenly probability is spread across those words). By introducing a "spike and slab" prior to separate these traits, the model achieves better predictive performance while generating significantly simpler and more interpretable topic sets.

The Catch-22 of Traditional HDP

In traditional nonparametric topic modeling using the Hierarchical Dirichlet Process (HDP), a single hyperparameter ($\gamma$) is forced to do two jobs at once. It controls both the sparsity of a topic (the number of terms with non-zero probability) and its smoothness (the uncertainty or "flatness" of the word distribution).

This "bundling" creates a mathematical catch-22. If you want sparse, interpretable topics, you must decrease $\gamma$. However, a small $\gamma$ also makes the model less "smooth," meaning it becomes overly confident in its word estimates and highly sensitive to noise. To compensate for this noise sensitivity, the HDP often enters a failure loop: it creates an excessive number of "junk" topics just to explain away infrequent words. The authors aim to break this link, allowing for topics that are both sparse and robustly smoothed.

Methods: The Sparse Topic Model (sparseTM)

To resolve this, the authors introduce the sparseTM, which modifies the generative process of the HDP by decoupling these parameters.

Graphical model of the sparseTM illustrating the decoupled sparsity and smoothness parameters.
  1. The "Spike and Slab" Prior: Instead of a standard Dirichlet distribution over the entire vocabulary, the model introduces a bank of binary Bernoulli variables ($b$) for every word in every topic.
  2. The "Spike" (Bernoulli variable) determines if a word is "on" or "off" for a specific topic.
  3. The "Slab" (Dirichlet distribution) then provides smoothing only for the words that are turned "on."
  4. Decoupled Parameters: Sparsity is now controlled by a Beta-distributed parameter ($\pi$), while smoothness is controlled independently by the Dirichlet parameter ($\gamma$).
  5. Inference: Because the number of possible word combinations is combinatorial ($2^V$), standard sampling is impossible. The authors developed an efficient collapsed Gibbs sampler that integrates out the topic distributions and term selectors, making the model computationally feasible.

Key Insights and Results

The model was evaluated on four real-world text collections: arXiv abstracts, Nematode Biology abstracts, NIPS full-text articles, and a collection of Computer Science conference abstracts.

By decoupling the parameters, sparseTM effectively ignores infrequent words (noise) without needing to invent entirely new topics to house them. This stops the HDP-LDA feedback loop, allowing a high $\gamma$ (high smoothing) even when the number of active words is low.

Comparison of HDP-LDA and STM across four datasets, showing STM achieves lower perplexity with fewer topics.
  • Predictive Performance: In 5-fold cross-validation, sparseTM consistently achieved lower predictive perplexity (a measure of how well the model predicts unseen data) compared to HDP-LDA across all four datasets.
  • Model Complexity: A good model shouldn't just predict well; it should do so with the fewest possible parameters. SparseTM produced much simpler models. For example, on the NIPS dataset, HDP-LDA often inferred over 1,400 topics, while sparseTM achieved better results with roughly 800 topics.
  • Qualitative Clarity: When looking at inferred topics, HDP-LDA often produced "noise topics" filled with unrelated, infrequent words (e.g., "epsilon," "stream," "motor"). SparseTM's topics remained more focused and semantically coherent.

Limitations and Future Work

The primary drawback of sparseTM is computational overhead. The authors note that the sparseTM Gibbs sampler is approximately 4 to 5 times slower than the standard HDP-LDA sampler due to the added complexity of sampling the selector variables. Additionally, while the model successfully decouples sparsity and smoothness, it introduces new hyperparameters for the Beta prior that may require tuning. Future work could involve applying these sparsity constraints to even larger datasets or exploring variational inference methods to bridge the speed gap.

Summary updated

Abstract

We present a nonparametric hierarchical Bayesian model of document collections that decouples sparsity and smoothness in the component distributions (i.e., the ``topics). In the sparse topic model (STM), each topic is represented by a bank of selector variables that determine which terms appear in the topic. Thus each topic is associated with a subset of the vocabulary, and topic smoothness is modeled on this subset. We develop an efficient Gibbs sampler for the STM that includes a general-purpose method for sampling from a Dirichlet mixture with a combinatorial number of components. We demonstrate the STM on four real-world datasets. Compared to traditional approaches, the empirical results show that STMs give better predictive performance with simpler inferred models.
BibTeX
@inproceedings{wang2009decoupling,
  author = {Chong Wang and David M. Blei},
  title = {{Decoupling Sparsity and Smoothness in the Discrete Hierarchical Dirichlet Process}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2009},
  url = {https://openreview.net/forum?id=BJbQCDbdZS},
}
Figure