The Discrete Infinite Logistic Normal Distribution

John Paisley, Chong Wang, David M. Blei

Abstract

We present the discrete infinite logistic normal distribution (DILN), a Bayesian nonparametric prior for mixed membership models. DILN is a generalization of the hierarchical Dirichlet process (HDP) that models correlation structure between the weights of the atoms at the group level. We derive a representation of DILN as a normalized collection of gamma-distributed random variables and study its statistical properties. We consider applications to topic modeling and derive a variational inference algorithm for approximate posterior inference. To deal with large-scale datasets, we also develop an online inference algorithm for DILN and compare its performance against online HDP and online LDA on the Nature magazine corpus, which contains approximately 350,000 articles.

Introduction

In probabilistic topic modeling, researchers often face a trade-off between the flexibility of nonparametrics and the descriptive power of correlation modeling. The Hierarchical Dirichlet Process (HDP) allows a model to automatically learn the number of topics from data but assumes these topics are independent. Conversely, the Correlated Topic Model (CTM) explicitly captures relationships between themes (e.g., "Genetics" and "DNA" co-occurring) but is parametric, requiring a fixed number of topics to be specified in advance.

The Discrete Infinite Logistic Normal (DILN) distribution bridges this gap. It provides a framework that is both nonparametric—allowing for an infinite potential number of topics—and capable of modeling complex correlation structures by embedding topics in a latent coordinate space.

Methodology

DILN extends the HDP by assigning every topic a location in a $d$-dimensional latent space. To determine the weight of a topic within a specific document, the model scales standard HDP weights by an exponentiated Gaussian Process (GP) evaluated at that topic's location. If two topics are positioned closely in this latent space, their weights will be highly correlated across the corpus.

Graphical model of the DILN distribution and its notation.

Mathematically, DILN is formulated as a normalized collection of gamma-distributed random variables. By utilizing the "rate" parameter of the gamma distribution, the model can inject correlation while maintaining a tractable structure for inference. This effectively "warps" the probability simplex of the HDP to account for spatial proximity between topics. For practical application, the authors derived a Mean-Field Variational Inference algorithm for batch processing and a Stochastic Variational Inference (SVI) version for massive datasets.

Experimental Results

The authors evaluated DILN against HDP and CTM using several text corpora, including Wikipedia, The New York Times, Science, and The Huffington Post. Across all batch evaluations, DILN consistently achieved lower held-out perplexity, indicating superior predictive accuracy.

Qualitatively, the model successfully identifies logical thematic structures. For example, in the Wikipedia dataset, the model found that the "Election" topic was positively correlated with "Parliament" and "Military," but negatively correlated with "Food and Drink."

Positive and negative correlation matrices discovered by DILN on the Wikipedia dataset.

In large-scale tests using the Nature magazine corpus (352,549 articles), the SVI-based DILN processed data significantly faster than batch methods. It reached a level of predictive performance that HDP and LDA could not match, even with extensive parameter tuning, demonstrating its scalability and robustness for real-world applications.

Conclusion and Future Work

DILN represents a significant step forward in mixed-membership modeling by combining the "infinite" flexibility of HDP with the structural insights of CTM. While the current implementation utilizes dot-product kernels and variational truncation for efficiency, future research may explore more complex kernel functions and truly truncation-free inference methods. Beyond text, DILN holds potential for other domains, such as modeling correlations between hidden state transitions in Hidden Markov Models (HMMs).

Summary updated

Abstract

We present the discrete infinite logistic normal distribution (DILN), a Bayesian nonparametric prior for mixed membership models. DILN is a generalization of the hierarchical Dirichlet process (HDP) that models correlation structure between the weights of the atoms at the group level. We derive a representation of DILN as a normalized collection of gamma-distributed random variables, and study its statistical properties. We consider applications to topic modeling and derive a variational inference algorithm for approximate posterior inference. We study the empirical performance of the DILN topic model on four corpora, comparing performance with the HDP and the correlated topic model (CTM). To deal with large-scale data sets, we also develop an online inference algorithm for DILN and compare with online HDP and online LDA on the Nature magazine, which contains approximately 350,000 articles.
BibTeX
@inproceedings{paisley2012discrete,
  author = {John Paisley and Chong Wang and David M. Blei},
  title = {{The Discrete Infinite Logistic Normal Distribution}},
  booktitle = {Bayesian},
  year = {2012},
  doi = {10.1214/12-ba734},
}
Figure