The Discrete Infinite Logistic Normal Distribution for Mixed-Membership Modeling

John W. Paisley, Chong Wang, David M. Blei

Introduction

The Discrete Infinite Logistic Normal (DILN) distribution, nicknamed "Dylan," is a novel Bayesian nonparametric prior designed to bridge the gap between two dominant paradigms in automated text analysis: the Hierarchical Dirichlet Process (HDP) and the Correlated Topic Model (CTM). While the HDP automatically learns the number of topics but assumes they are independent, the CTM captures topic correlations but requires a pre-specified number of topics ($K$). DILN provides a unified framework that is both infinite (nonparametric) and capable of modeling complex topic correlations, leading to better predictive performance and more intuitive document summaries.

Problem Statement

Existing models face a significant trade-off between flexibility and representational power. The HDP's independence assumption prevents it from representing the fact that certain topics (e.g., "politics" and "economics") are more likely to co-occur than others. Conversely, the CTM's finite nature leads to overfitting or under-summarization if the number of topics is incorrectly guessed. Previous attempts to create an "infinite logistic normal" distribution failed because they did not produce the "sparse" distributions necessary for effective topic sharing across documents. The authors address this by building a model that retains the sharing property of the HDP while adding the co-occurrence logic of the CTM.

The DILN Model

DILN is conceptualized as a "scaled HDP." The core innovation involves placing topics within a latent space where correlations are determined by spatial distance, modeled via a Gaussian Process (GP).

Graphical model and notation for the DILN distribution.

To make the mathematics tractable, the authors utilize a normalized gamma representation. Topic weights are drawn from Gamma distributions where the rate parameter is controlled by the GP. This clever technical pivot allows the correlation structure to be absorbed into a standard distribution. Inference is performed using a Variational Bayes algorithm that is backward compatible; by setting specific parameters to one, it effectively becomes an inference algorithm for the standard HDP. Efficiency is further improved by optimizing the kernel matrix directly, avoiding expensive matrix inversions and speeding up the learning process.

Experimental Evaluation

The authors evaluated DILN on four diverse text corpora: Huffington Post (4,000 documents), New York Times (7,000 documents), Science (7,000 documents), and Wikipedia (7,000 documents). DILN was compared against the standard HDP and several CTM configurations with varying topic counts ($K=20, 50, 150$).

Key Results and Insights

DILN consistently outperformed both HDP and CTM across all datasets in terms of held-out perplexity, indicating superior predictive performance for unseen documents.

Held-out perplexity comparison across four corpora showing DILN's superior performance and robustness.

Key findings from the evaluation include:

  • Superior Perplexity: DILN achieved lower (better) held-out perplexity than both the HDP and the CTM in all four datasets.
  • Robustness to Hyperparameters: The model outperformed competitors across a wide range of Dirichlet hyperparameter (gamma) settings, suggesting it is less sensitive to manual tuning.
  • Sparsity and Sharing: Unlike previous attempts at infinite logistic normals, DILN maintains sparsity. Even though the model could theoretically use infinite topics, the variational approximation prefers a small, relevant subset (typically 50–100 topics), preventing overfitting.
  • Interpretability: By learning a kernel matrix, DILN allows for the visualization of topic correlations in a 2D map. Topics that are "close" in this map (such as "police" and "prison") are those the model has learned tend to appear together in the same documents.
  • Computational Efficiency: Despite the added complexity of modeling correlations, DILN’s computational time was comparable to the HDP, averaging roughly one minute per iteration on the tested hardware.

Conclusion and Future Work

DILN represents a significant advancement in Bayesian nonparametrics, offering a general prior for mixed-membership modeling that captures dependencies without sacrificing the flexibility of infinite topic discovery. While current inference uses a truncation level approximation ($T=200$), the model remains theoretically infinite. Future work may explore more sophisticated kernels to capture nuanced hierarchies and apply DILN to other domains such as population genetics, image analysis, or social network modeling.

Summary updated

BibTeX
@inproceedings{paisley2012discrete,
  author = {John W. Paisley and Chong Wang and David M. Blei},
  title = {{The Discrete Infinite Logistic Normal Distribution for Mixed-Membership Modeling}},
  booktitle = {Bayesian},
  year = {2012},
  doi = {10.1214/12-ba734},
}
Figure