Abstract
This paper concerns the introduction of a new Markov Chain Monte Carlo scheme for posterior sampling in Bayesian nonparametric mixture models with priors that belong to the general Poisson-Kingman class. We present a novel compact way of representing the infinite dimensional component of the model such that while explicitly representing this infinite component it has less memory and storage requirements than previous MCMC schemes. We describe comparative simulation results demonstrating the efficacy of the proposed MCMC algorithm against existing marginal and conditional MCMC samplers.
Introduction
In Bayesian Nonparametrics (BNP), models are designed to grow in complexity as more data is gathered, often utilizing clustering models where the number of clusters is not fixed a priori. However, performing inference—specifically sampling from the posterior—on these infinite-dimensional models is computationally challenging. Historically, researchers have relied on two main strategies: marginal samplers, which are memory-efficient but suffer from poor mixing, and conditional samplers, which improve mixing through truncation but require significant memory and storage.
This paper introduces a "hybrid" MCMC sampler for Poisson-Kingman (PK) mixture models. This approach combines the memory efficiency of marginal samplers with the superior mixing properties of conditional samplers by explicitly representing only the occupied clusters and summarizing the infinite "tail" of the model into a single surplus mass variable.
Methodology
The proposed hybrid sampler is based on a "varying table size Chinese restaurant representation." The core innovation lies in the Surplus Mass Representation. Instead of truncating the infinite tail at an arbitrary point, the model is represented using the weights of currently occupied clusters plus a single random variable $V$ representing the "surplus mass" of all unoccupied clusters.
Key technical features include:
- Lazy Sampling: Cluster weights are only instantiated when an observation is assigned to a new cluster, keeping the state space minimal.
- Complete Conditionals: The authors derive exact mathematical updates for the surplus mass and cluster weights within a Gibbs sampling framework.
- Generality: Unlike many MCMC methods focused on specific processes like the Dirichlet Process, this hybrid approach is a general-purpose tool designed for the entire Poisson-Kingman family.
Experimental Results
The hybrid sampler was evaluated using synthetic data across various processes (Pitman-Yor, Normalized Stable, etc.) and the classic Roeder’s Galaxy dataset. Performance was measured using Effective Sample Size (ESS), which quantifies the number of independent samples produced per unit of time.
As shown in the comparative results, the hybrid sampler consistently outperforms both marginal and conditional counterparts. For instance, in the Normalized Stable process ($\sigma=0.5$), the hybrid sampler achieved an ESS of approximately 4877, significantly higher than the ~912 achieved by the conditional sampler. Furthermore, the hybrid approach demonstrated new capabilities by successfully performing inference on the "-logBeta" process, for which no other known sampler currently exists.
Discussion and Future Work
While the hybrid sampler offers significant improvements in speed and accuracy, some limitations remain. In the "Hybrid-MH" variant used for intractable mass densities, the Metropolis-Hastings acceptance rates are not always optimal, suggesting that future research into better proposal distributions is warranted. Additionally, performance remains sensitive to hyperparameters like $\sigma$. Future work will focus on investigating more efficient ways to update the surplus mass to further reduce autocorrelation in the Markov chain and move closer to fully automated inference in probabilistic programming environments.
Summary updated