Reliable and Scalable Variational Inference for the Hierarchical Dirichlet Process

Michael C. Hughes, Dae Il Kim, Erik B. SudderthAISTATS 2015 · OpenReview

Abstract

This paper introduces a new variational inference framework for Hierarchical Dirichlet Process (HDP) models that is both scalable to millions of documents and reliable in determining the optimal number of topics. By transitioning from point estimates to full variational posteriors and employing a novel surrogate likelihood bound, the authors enable dynamic "merge and delete" moves. These moves automatically prune redundant or irrelevant topics, resulting in more compact and interpretable models than previous state-of-the-art methods.

Introduction

The Hierarchical Dirichlet Process (HDP) is a powerful nonparametric tool for topic modeling, capable of learning the number of latent themes as data grows. However, HDPs are notoriously difficult to train. Existing methods suffer from a "trilemma": MCMC samplers are too slow for large datasets, stochastic variational methods often get trapped in poor local optima (splitting single topics into redundant ones), and point-estimate approximations lead to "topic bloat," where the model incorrectly assumes that adding empty topics always improves the fit.

Methods: A Reliable Variational Framework

The authors propose three key innovations to the variational inference (VI) pipeline to create an algorithm that scales like stochastic methods while retaining the model-selection reliability of slower samplers.

Model Architecture and Representation

By using a "direct assignment" representation rather than the traditional "Chinese Restaurant Franchise" representation, local updates become linear relative to the number of topics. This significantly increases speed and allows the model to be structured across local, summary, and global levels for efficient processing.

Hierarchical structure and plate notation for the proposed variational HDP model.

Full Variational Posteriors and Surrogate Bounds

Instead of using a single "best guess" point estimate for global topic frequencies, the authors define a full Beta distribution posterior. This captures uncertainty and allows the model to naturally penalize empty topics. To handle the resulting non-conjugacy, they derive a novel surrogate likelihood bound on the Dirichlet cumulant function, creating an objective that is easy to optimize while remaining a valid lower bound on the true evidence.

Key Insights: Overcoming Topic Bloat

A critical finding is that point-estimating global weights is "pathological" because it rewards the model for adding empty topics. As shown in the analysis, the proposed surrogate bound correctly identifies that simpler models are often better, mirroring the behavior of the exact HDP objective.

Validation of the surrogate likelihood bound and its effect on topic selection.

Scalability and Global Moves

To escape local optima, the algorithm periodically attempts to merge highly correlated topics or delete "junk" topics with minimal data assignment. These global moves act as a search mechanism that gradient-based optimization lacks. The algorithm is implemented using memoized variational inference, allowing it to process data in batches and scale to massive datasets like the NYTimes corpus.

Key Results

  • Compactness: On Wikipedia datasets, the proposed method reduced an initial 200 topics to fewer than 100 while improving held-out likelihood, whereas competitors produced lower-quality, redundant topics.
  • Speed and Scale: On a dataset of 1.8 million NYTimes articles, the memoized algorithm completed eight full passes in the time it took previous state-of-the-art methods to complete just one.
  • Image Modeling: The HDP admixture outperformed standard mixture models for image patches by adapting topic weights to specific image content (e.g., favoring smooth topics for sky images).

Conclusion

While the current implementation relies on some heuristic thresholds for "delete" moves, the surrogate bound approach provides a principled foundation for reliable nonparametric inference. Future work may extend this framework to other HDP-based structures, such as relational or sequential models, further broadening the reach of scalable, reliable Bayesian modeling.

Summary updated

Abstract

We introduce a new variational inference objective for hierarchical Dirichlet process admixture models. Our approach provides novel and scalable algorithms for learning nonparametric topic models o...
BibTeX
@inproceedings{hughes2015reliable,
  author = {Michael C. Hughes and Dae Il Kim and Erik B. Sudderth},
  title = {{Reliable and Scalable Variational Inference for the Hierarchical Dirichlet Process}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2015},
  url = {https://openreview.net/forum?id=aEM-pxz_HQY},
}

Loading related content...

Figure