Abstract
We present the discrete infinite logistic normal distribution (DILN), a Bayesian nonparametric prior for mixed membership models. DILN is a generalization of the hierarchical Dirichlet process (HDP) that models correlation structure between the weights of the atoms at the group level. We derive a representation of DILN as a normalized collection of gamma-distributed random variables and study its statistical properties. We consider applications to topic modeling and derive a variational inference algorithm for approximate posterior inference. To deal with large-scale datasets, we also develop an online inference algorithm for DILN and compare its performance against online HDP and online LDA on the Nature magazine corpus, which contains approximately 350,000 articles.
Introduction
In probabilistic topic modeling, researchers often face a trade-off between the flexibility of nonparametrics and the descriptive power of correlation modeling. The Hierarchical Dirichlet Process (HDP) allows a model to automatically learn the number of topics from data but assumes these topics are independent. Conversely, the Correlated Topic Model (CTM) explicitly captures relationships between themes (e.g., "Genetics" and "DNA" co-occurring) but is parametric, requiring a fixed number of topics to be specified in advance.
The Discrete Infinite Logistic Normal (DILN) distribution bridges this gap. It provides a framework that is both nonparametric—allowing for an infinite potential number of topics—and capable of modeling complex correlation structures by embedding topics in a latent coordinate space.
Methodology
DILN extends the HDP by assigning every topic a location in a $d$-dimensional latent space. To determine the weight of a topic within a specific document, the model scales standard HDP weights by an exponentiated Gaussian Process (GP) evaluated at that topic's location. If two topics are positioned closely in this latent space, their weights will be highly correlated across the corpus.
Mathematically, DILN is formulated as a normalized collection of gamma-distributed random variables. By utilizing the "rate" parameter of the gamma distribution, the model can inject correlation while maintaining a tractable structure for inference. This effectively "warps" the probability simplex of the HDP to account for spatial proximity between topics. For practical application, the authors derived a Mean-Field Variational Inference algorithm for batch processing and a Stochastic Variational Inference (SVI) version for massive datasets.
Experimental Results
The authors evaluated DILN against HDP and CTM using several text corpora, including Wikipedia, The New York Times, Science, and The Huffington Post. Across all batch evaluations, DILN consistently achieved lower held-out perplexity, indicating superior predictive accuracy.
Qualitatively, the model successfully identifies logical thematic structures. For example, in the Wikipedia dataset, the model found that the "Election" topic was positively correlated with "Parliament" and "Military," but negatively correlated with "Food and Drink."
In large-scale tests using the Nature magazine corpus (352,549 articles), the SVI-based DILN processed data significantly faster than batch methods. It reached a level of predictive performance that HDP and LDA could not match, even with extensive parameter tuning, demonstrating its scalability and robustness for real-world applications.
Conclusion and Future Work
DILN represents a significant step forward in mixed-membership modeling by combining the "infinite" flexibility of HDP with the structural insights of CTM. While the current implementation utilizes dot-product kernels and variational truncation for efficiency, future research may explore more complex kernel functions and truly truncation-free inference methods. Beyond text, DILN holds potential for other domains, such as modeling correlations between hidden state transitions in Hidden Markov Models (HMMs).
Summary updated