Abstract
Many practical modeling problems involve discrete data that are best represented as draws from multinomial or categorical distributions. For example, nucleotides in a DNA sequence, children's names in a given state and year, and text documents are all commonly modeled with multinomial distributions. In all of these cases, we expect some form of dependency between the draws: the nucleotide at one position in the DNA strand may depend on the preceding nucleotides, children's names are highly correlated from year to year, and topics in text may be correlated and dynamic. These dependencies are not naturally captured by the typical Dirichlet-multinomial formulation. Here, we leverage a logistic stick-breaking representation and recent innovations in Pólya-gamma augmentation to reformulate the multinomial distribution in terms of latent variables with jointly Gaussian likelihoods, enabling us to take advantage of a host of Bayesian inference techniques for Gaussian models with minimal overhead.
Introduction: The Challenge of Dependent Discrete Data
In fields ranging from natural language processing to genomics and social sciences, researchers frequently encounter discrete count data—such as word frequencies in documents, nucleotide sequences in DNA, or the popularity of names over time. Traditionally, these data types are modeled using the Dirichlet-Multinomial framework. However, the Dirichlet distribution is inherently "correlation-blind"; it assumes independence between categories that are often naturally linked. For instance, a document discussing "physics" is statistically more likely to also mention "gravity," a dependency that the standard Dirichlet prior cannot easily represent.
To capture these correlations, researchers often turn to latent Gaussian structures, such as Gaussian Processes (GPs) or Linear Dynamical Systems (LDS). The primary obstacle is that Gaussian priors and multinomial likelihoods are non-conjugate. This lack of conjugacy typically necessitates slow, complex inference algorithms like Particle MCMC or specialized variational approximations that are difficult to scale and implement.
Methodology: Bridging Gaussians and Multinomials
This paper introduces a clever mathematical bridge that allows researchers to model discrete data using the powerful toolkit of Gaussian inference. The approach simplifies dependent multinomial modeling through two key innovations:
- Logistic Stick-Breaking: Instead of the standard symmetric "softmax" (logistic-normal) function, the authors decompose a $K$-way multinomial choice into a sequence of $K-1$ binary decisions. This recursive "stick-breaking" process is just as expressive as the softmax but significantly more amenable to Bayesian inference.
- Pólya-Gamma (PG) Augmentation: By introducing auxiliary Pólya-Gamma random variables, the authors use a mathematical identity to express the logistic likelihood as an integral over a Gaussian kernel. This transformation makes the multinomial likelihood conditionally Gaussian.
Once conditioned on the auxiliary variables, the entire model behaves like a standard Gaussian model. This modularity allows researchers to leverage existing high-performance software for Kalman filters, smoothing, or Gaussian Processes to handle discrete data with minimal additional overhead.
Empirical Evaluation and Key Results
The authors demonstrate the versatility and efficiency of their approach across several distinct domains, including text corpora (AP News, 20 Newsgroups), genomics (human DNA sequences), and demographics (US Census baby names).
Efficiency in Correlated Topic Modeling
In text analysis, the stick-breaking Correlated Topic Model (SB-CTM) demonstrated superior performance. As shown in the figure below, the model effectively captures complex topic correlations while converging to high predictive log-likelihoods significantly faster than traditional Variational Bayes (VB) or Gibbs sampling approaches used for the standard Logistic-Normal CTM.
Spatiotemporal Modeling of Demographic Trends
The framework's ability to handle complex dependencies was further tested on US Census data tracking children's names from 1910 to 2013. By embedding a Gaussian Process prior within the stick-breaking framework, the model successfully captured spatiotemporal trends. It accurately modeled how name popularities, such as "Ethan," shift geographically and over time, outperforming baseline models that struggle with sparse data or complex spatial correlations.
Discussion
The primary advantage of this framework is its modularity. By transforming multinomial problems into conditionally Gaussian ones, it overcomes the "softmax bottleneck" that has long hindered Bayesian inference for discrete data. While the recursive nature of stick-breaking introduces an ordering sensitivity among categories, the authors argue that this does not limit representational capacity and is a small price to pay for the resulting computational gains. Future work may explore extending this unified Gaussian latent space to handle "mixed" data types, combining counts, continuous values, and binary data in a single coherent model.
Summary updated