TL;DR
This paper unifies count and mixture modeling under a novel Negative Binomial (NB) process framework. By leveraging a gamma process prior on a Poisson process rate measure, the authors derive a flexible NB process that naturally handles both count data and the assignment of data points to clusters, offering theoretical and computational advantages over existing methods like the Dirichlet process.
What Problem Does This Paper Address?
Traditional mixture modeling, which assigns data points to clusters, often treats the number of points per cluster as a count. However, this problem is typically framed within a Dirichlet-multinomial framework, which can be inflexible. Simultaneously, count data often exhibits overdispersion (variance exceeding the mean), making standard Poisson models inadequate. The authors aim to bridge this gap by developing a unified framework that can model both the overdispersed counts of data points within clusters and the underlying mixture structure, offering a more principled and flexible approach. They specifically address the limitations of the Dirichlet process, which has a single parameter controlling both variability and the number of distinct components, and the challenges in inferring parameters for overdispersed count models like the Negative Binomial distribution.
Methods
The core of the paper lies in constructing the Negative Binomial (NB) process. This is achieved by:
- Gamma Process Prior on Poisson Rate: A gamma process is used to model the rate measure of a Poisson process.
- Normalization for Mixture Modeling: Normalizing this gamma process yields a random probability measure suitable for mixture modeling, which is not necessarily a Dirichlet process.
- Marginalization for Count Modeling: Marginalizing out the gamma process leads to an NB process for count modeling.
- Poisson-Logarithmic Bivariate Distribution: A novel bivariate distribution is introduced, connecting the Poisson, logarithmic, NB, and Chinese restaurant table distributions. This is crucial for joint inference and understanding model properties.
- Data Augmentation and Marginalization: The authors develop techniques to augment the NB process, allowing it to be related to gamma-Poisson and compound Poisson representations. This facilitates efficient Bayesian inference.
- Hierarchical Construction: The framework is extended to hierarchical models, allowing for sharing of statistical strength across groups, leading to connections with the Dirichlet Process (DP) and Hierarchical Dirichlet Process (HDP).
- NB Process Family: A variety of NB processes are constructed by varying how dispersion and probability parameters are shared across groups, including beta-geometric, beta-NB, marked-beta-NB, marked-gamma-NB, and zero-inflated-NB processes.
Datasets
The paper uses the "Psychological Review corpus" for evaluating topic modeling applications. This corpus consists of 1281 abstracts with a vocabulary of 2566 terms and 71,279 total word counts. The authors describe experiments where they vary the percentage of words used for training and testing.
Key Insights and Ideas
- Unifying Count and Mixture Modeling: The NB process elegantly bridges the gap between modeling overdispersed counts and performing mixture modeling, a significant theoretical contribution.
- Flexibility Beyond Dirichlet Process: The NB process, and its extensions, offer more flexibility than the standard Dirichlet process by allowing for separate control over dispersion and probability parameters.
- The Power of Gamma-Poisson Construction: This construction is key to uniting count and mixture modeling and enables efficient Bayesian inference through data augmentation.
- Poisson-Logarithmic Distribution as a Bridge: This bivariate distribution is a critical tool for connecting various count-based distributions and understanding the NB process.
- Inferring Dispersion is Crucial: Experimental results highlight that inferring both the NB dispersion and probability parameters is essential for good performance, especially in applications like topic modeling.
- NB Processes as Generalizations: The NB process and its variants can be seen as generalizations of DP and HDP, offering richer modeling capabilities.
Key Results
- Efficient Bayesian Inference: The proposed data augmentation and marginalization techniques lead to efficient closed-form conditional posteriors and predictive distributions.
- Connections to Existing Models: The NB process and gamma-NB process are shown to be reducible to the Dirichlet process and HDP, respectively, under certain conditions.
- Variety of NB Processes: The paper constructs and analyzes a family of NB processes with distinct sharing mechanisms, offering a rich toolkit for various applications.
- Topic Modeling Performance: Experimental results on the Psychological Review corpus demonstrate that the proposed NB process models, particularly the Gamma-NB process, can achieve competitive or superior performance compared to LDA and CRF-HDP, especially when inferring parameters. The Gamma-NB process was shown to discover more active topics than the Beta-NB process.
Limitations and Future Work
- Computational Complexity: While efficient inference is derived, the paper notes that representing an infinite number of atoms is infeasible in practice. Block Gibbs sampling with a truncated number of atoms is used, but this can lead to increasing computational cost with larger truncation levels.
- Slice Sampling for NB Processes: The authors suggest that investigating slice sampling for NB process models could be beneficial, as it might offer more amenable likelihoods for posterior simulation without imposing normalization on atom weights.
- Collapsed Inference: Developing collapsed inference schemes for NB processes is identified as a future direction, which would require well-defined prediction rules and conjugacy between likelihoods and priors.
- Zero-Inflated Models: While the NB-FTM model is introduced to handle excessive zeros, the paper notes that more direct approaches using $p_k$ or $p_j$ often yield better results, suggesting that the utility of zero-inflated models might depend on specific data characteristics (e.g., scarce training words).
- Parameter Inference for HDP: The paper points out that inferring the concentration parameters of HDP can be challenging, especially for discrete base measures, whereas the proposed NB process framework offers analytic conditional posteriors for similar parameters.
Summary updated