The Discrete Infinite Logistic Normal Distribution

John Paisley, Chong Wang, David M. Blei

Abstract

We present the discrete infinite logistic normal distribution (DILN), a Bayesian nonparametric prior for mixed membership models. DILN generalizes the hierarchical Dirichlet process (HDP) to model correlation structure between the weights of the atoms at the group level. We derive a representation of DILN as a normalized collection of gamma-distributed random variables and study its statistical properties. We derive a variational inference algorithm for approximate posterior inference. We apply DILN to topic modeling of documents and study its empirical performance on four corpora, comparing performance with the HDP and the correlated topic model (CTM). To compute with large-scale data, we develop a stochastic variational inference algorithm for DILN and compare with similar algorithms for HDP and latent Dirichlet allocation (LDA) on a collection of 350,000 articles from Nature.

Introduction: Beyond Independence in Topic Modeling

In probabilistic topic modeling, researchers have traditionally faced a trade-off between model flexibility and structural complexity. The Hierarchical Dirichlet Process (HDP) allows for an automatically discovered, "infinite" number of topics (Bayesian nonparametrics) but assumes these topics are independent. Conversely, the Correlated Topic Model (CTM) captures how topics relate to one another—recognizing that a document about "Elections" is more likely to discuss "Politics" than "Marine Biology"—but requires a fixed, pre-defined number of topics.

This paper introduces the Discrete Infinite Logistic Normal (DILN) distribution (pronounced "Dylan"). DILN provides a "best of both worlds" approach, bringing correlation modeling to the flexible world of Bayesian nonparametrics. By combining the HDP with Gaussian Processes, DILN allows models to learn an infinite number of topics while simultaneously capturing their complex co-occurrence patterns.

Methodology

The core innovation of DILN lies in its formulation as a normalized collection of gamma-distributed random variables. The authors achieve this through two primary mathematical representations:

  1. Scaled HDP: A standard HDP is scaled using an exponentiated Gaussian Process (GP).
  2. Normalized Gamma Process: The second parameter of the gamma distribution is utilized to model the underlying covariance structure.

To implement this, each topic is assigned a location in a latent $d$-dimensional space. The correlation between any two topics is determined by the proximity of their latent locations. To handle the resulting mathematical complexity, the authors developed a "mean-field" variational inference algorithm. Furthermore, to scale to massive datasets, they adapted this into a Stochastic Variational Inference (SVI) algorithm, which uses Taylor expansions and auxiliary variables to maintain efficiency in non-conjugate settings.

Experimental Setup and Datasets

The authors evaluated DILN across several diverse text corpora to test both predictive accuracy and scalability:

  • Standard Corpora: The Huffington Post (3k documents), New York Times (5k), Science (5k), and Wikipedia (5k).
  • Large-Scale Corpus: A massive archive of 352,549 articles from Nature (1869–2003), used to benchmark the SVI approach against HDP and Latent Dirichlet Allocation (LDA).
Table of 30 topics discovered by the model, showing top keywords for each.

Key Results and Insights

DILN consistently outperformed both HDP and CTM in terms of perplexity, a standard metric for how well a model predicts held-out data. On the large-scale Nature dataset, DILN reached higher predictive performance than its competitors, proving that the added complexity of correlation modeling does not hinder scalability when paired with SVI.

Beyond numerical performance, the model learns intuitive semantic relationships. By mapping topics to a latent space, DILN identifies meaningful positive and negative correlations. For example, as shown in the topic list above, Topic 3 ("Party/Election") shows a strong positive correlation with Topic 12 ("Constitution/Parliament") but a negative correlation with Topic 20 ("Food/Meat/Drink").

Visual representation of the learned positive (green) and negative (red) correlation matrices between 30 topics.

Conclusion and Future Work

DILN demonstrates that correlation is a powerful predictive signal in nonparametric models. While the current implementation primarily uses dot-product kernels for computational tractability, future work could explore richer structures like RBF kernels. Additionally, the DILN framework is not limited to text; it could be adapted for Hidden Markov Models (HMMs) to allow for an infinite number of states with correlated transition probabilities, opening new doors for time-series analysis and sequence modeling.

Summary updated

Abstract

We present the discrete infinite logistic normal distribution (DILN), a Bayesian nonparametric prior for mixed membership models. DILN generalizes the hierarchical Dirichlet process (HDP) to model correlation structure between the weights of the atoms at the group level. We derive a representation of DILN as a normalized collection of gamma-distributed random variables and study its statistical properties. We derive a variational inference algorithm for approximate posterior inference. We apply DILN to topic modeling of documents and study its empirical performance on four corpora, comparing performance with the HDP and the correlated topic model (CTM). To compute with large-scale data, we develop a stochastic variational inference algorithm for DILN and compare with similar algorithms for HDP and latent Dirichlet allocation (LDA) on a collection of 350,000 articles from Nature.
BibTeX
@inproceedings{paisley2012discrete,
  author = {John Paisley and Chong Wang and David M. Blei},
  title = {{The Discrete Infinite Logistic Normal Distribution}},
  booktitle = {Bayesian},
  year = {2012},
  doi = {10.1214/12-ba734},
}
Figure