A Marginal Sampler for σ-Stable Poisson–Kingman Mixture Models

María Lomelí, Stefano Favaro, Yee Whye TehJournal of Computational and Graphical Statistics 2016 · DOI · arXiv

Abstract

We investigate the class of σ-stable Poisson–Kingman random probability measures (RPMs) in the context of Bayesian nonparametric mixture modeling. This is a large class of discrete RPMs, which encompasses most of the popular discrete RPMs used in Bayesian nonparametrics, such as the Dirichlet process, Pitman–Yor process, the normalized inverse Gaussian process, and the normalized generalized Gamma process. We show how certain sampling properties and marginal characterizations of σ-stable Poisson–Kingman RPMs can be usefully exploited for devising a Markov chain Monte Carlo (MCMC) algorithm for performing posterior inference with a Bayesian nonparametric mixture model. Specifically, we introduce a novel and efficient MCMC sampling scheme in an augmented space that has a small number of auxiliary variables per iteration. We apply our sampling scheme to a density estimation and clustering tasks with unidimensional and multidimensional datasets, and compare it against competing MCMC sampling schemes.

Introduction

In Bayesian nonparametrics (BNP), researchers often seek to model data without pre-specifying the number of clusters, allowing the model complexity to grow naturally with the data. While the Dirichlet Process (DP) is the standard choice for such tasks, it can be overly restrictive in its clustering behavior. The σ-stable Poisson–Kingman (PK) class offers a much more flexible alternative, acting as a "super-family" that includes the Pitman–Yor process, the normalized inverse Gaussian process, and the normalized generalized Gamma (NGG) process.

Despite their theoretical appeal, σ-stable PK models have historically faced a significant computational barrier: their probability density functions are often intractable, lacking simple closed-form expressions and requiring expensive numerical integration. This paper introduces a novel marginal MCMC sampler that overcomes these hurdles, making these powerful models significantly more accessible.

The Challenge of Intractability

Previous attempts to perform inference with σ-stable PK models relied on conditional samplers. These methods maintain an explicit representation of the infinite-dimensional random measure. However, conditional samplers often suffer from high memory usage and slow "mixing" (the speed at which the algorithm explores the posterior) because they require a fluctuating number of auxiliary variables that can grow with the size of the dataset.

To address this, the authors propose a marginal sampler. Instead of tracking the infinite-dimensional measure, they integrate it out, focusing exclusively on how data points are partitioned into clusters. This approach significantly reduces the complexity of the state space.

Technical Innovations

The efficiency of the new sampler is driven by three key innovations:

  1. Zolotarev’s Representation: The authors leverage a specific integral representation of σ-stable densities to introduce a new auxiliary variable ($Z$). This transforms a mathematically difficult density into a manageable sampling problem, bypassing the need for numerical integration entirely.
  2. Data Augmentation: The scheme introduces three auxiliary variables ($W, R, Z$) that decouple the dependencies between the total mass of the process and the cluster assignments.
  3. Fixed Complexity: Unlike conditional samplers, this marginal approach uses a fixed number of auxiliary variables per iteration, regardless of the dataset size, leading to more predictable and efficient performance.

Furthermore, the algorithm is designed to work with non-conjugate models—where the prior for cluster parameters and the data likelihood are not mathematically compatible—by extending the "Reuse" algorithm strategy.

Experimental Results

The authors evaluated their sampler on two classic benchmarks to test its density estimation and clustering capabilities:

  • Galaxy Dataset (Unidimensional): 82 measurements of galaxy velocities. The sampler effectively identified "clumps" in the data, a standard test for density estimation.
  • Olive Oil Dataset (Multidimensional): 120 samples of triacylglyceride profiles. After reducing the 4,000 dimensions to 8 using PCA, the sampler successfully identified primary clusters and distinguished between different varieties of vegetable oils.

Performance Gains

In head-to-head comparisons with state-of-the-art conditional samplers, the marginal sampler demonstrated superior efficiency. For instance, when applied to a Pitman–Yor process, the marginal sampler achieved an Effective Sample Size (ESS) of approximately 12,792, compared to only 707 for the conditional version in a similar timeframe.

Additionally, the authors found that the Normalized Generalized Gamma (NGG) process slightly outperformed the standard DP and Pitman–Yor models in terms of predictive probability, justifying the use of this more complex class of models for real-world data.

Conclusion

This work provides a unified and efficient framework for performing posterior inference across the entire σ-stable Poisson–Kingman family. By utilizing Zolotarev’s representation to avoid numerical bottlenecks, the authors have created a tool that is both faster and more robust than previous methods. This sampler serves as a vital building block for more complex hierarchical models in fields ranging from genomics to topic modeling.

Summary updated

Abstract

We investigate the class of σ-stable Poisson–Kingman random probability measures (RPMs) in the context of Bayesian nonparametric mixture modeling. This is a large class of discrete RPMs, which encompasses most of the popular discrete RPMs used in Bayesian nonparametrics, such as the Dirichlet process, Pitman–Yor process, the normalized inverse Gaussian process, and the normalized generalized Gamma process. We show how certain sampling properties and marginal characterizations of σ-stable Poisson–Kingman RPMs can be usefully exploited for devising a Markov chain Monte Carlo (MCMC) algorithm for performing posterior inference with a Bayesian nonparametric mixture model. Specifically, we introduce a novel and efficient MCMC sampling scheme in an augmented space that has a small number of auxiliary variables per iteration. We apply our sampling scheme to a density estimation and clustering tasks with unidimensional and multidimensional datasets, and compare it against competing MCMC sampling schemes. Supplementary materials for this article are available online.
BibTeX
@article{lomeli2016marginal,
  author = {María Lomelí and Stefano Favaro and Yee Whye Teh},
  title = {{A Marginal Sampler for σ-Stable Poisson–Kingman Mixture Models}},
  journal = {Journal of Computational and Graphical Statistics},
  year = {2016},
  doi = {10.1080/10618600.2015.1110526},
}
Figure