Abstract
Bayesian statistical models allow us to formalise our knowledge about the world and reason about our uncertainty, but there is a need for better procedures to accurately encode its complexity. One way to do so is through compositional models, which are formed by combining blocks consisting of simpler models. This thesis expands the choice of Bayesian nonparametric (BNP) blocks for constructing tractable compositional models. While most BNP models use a Dirichlet Process or a Pitman-Yor process, this work shows how to overcome intractabilities to obtain analogous representations for the broader class of Poisson-Kingman priors. The main contribution is a variety of exact inference schemes—Markov chain Monte Carlo and Sequential Monte Carlo methods—that target the true posterior, providing general-purpose tools for probabilistic programming.
Introduction
Bayesian Nonparametrics (BNP) is a powerful framework for clustering where the number of clusters ($k$) is not fixed but grows with the data. While the Dirichlet Process (DP) and Pitman–Yor (PY) process are widely used due to their tractable representations, they represent only a small subset of the potential models. The broader family of Poisson–Kingman (PK) priors offers significantly more flexibility in capturing diverse clustering behaviors, yet they have remained underutilized due to computational intractability.
This research unlocks the PK family by providing novel, exact inference algorithms that make these flexible models computationally practical for complex, real-world datasets.
The Challenge: Beyond the Dirichlet Process
In standard mixture modeling, pre-specifying the number of components often leads to overfitting or underfitting. BNP models address this by assuming an infinite number of potential components. However, inference in the general PK family is notoriously difficult because their probability densities often involve intractable integrals. Previous research often relied on crude approximations or restricted themselves to the simpler DP/PY cases. This work develops exact inference schemes that target the true posterior without systematic bias, turning numerical integration challenges into standard sampling problems.
Methodology
Three main computational contributions are introduced to facilitate inference in PK models:
- The Marginal Sampler: This approach integrates out the infinite-dimensional component. For $\sigma$-stable PK priors, a data augmentation trick based on the Zolotarev integral representation is employed to handle intractable densities.
- The Hybrid Sampler: This novel MCMC scheme combines the strengths of marginal and conditional samplers. It represents the model using the weights of occupied clusters and a single "surplus mass" term for all unoccupied clusters. This approach is memory-efficient and avoids the truncation errors typical of slice samplers.
- Sequential Monte Carlo (SMC): SMC schemes are developed for sequential inference and estimating the marginal likelihood. This enables formal Bayesian hypothesis testing to compare infinite models against finite alternatives.
Experimental Evaluation
The proposed methods were validated on two primary datasets:
- Galaxy Dataset: A benchmark of 82 galaxy velocities used to test clustering efficiency.
- Oil Profiles Dataset: A high-dimensional dataset of 120 triacylglyceride profiles (reduced via PCA). The goal was to cluster oil types (olive, vegetable, blends) without prior knowledge of the number of subtypes.
Key Results
- Efficiency: The Hybrid Sampler significantly outperformed existing conditional slice samplers. On the Galaxy dataset, it achieved a much higher Effective Sample Size (ESS) per second, demonstrating superior exploration of the posterior landscape.
- Stability: For specific parameter ranges (e.g., $\sigma=0.5$), the hybrid sampler proved more numerically stable than previous methods.
- Real-World Utility: On the oil dataset, PK-based mixture models successfully recovered meaningful clusters and provided better predictive probabilities than the standard DP, proving that the added flexibility of PK priors translates to better model performance.
Discussion and Future Work
The "surplus mass" concept in the hybrid sampler allows for an exact representation of an infinite-dimensional model using finite memory. By providing a general-purpose framework for PK priors, this work moves BNP closer to the "probabilistic programming" ideal.
However, challenges remain regarding scalability to datasets with millions of points. Future work may involve extending mathematical identities to the full range of PK priors and incorporating "split-merge" proposals to further improve MCMC mixing speeds in complex clustering configurations.
This summary is based on the PhD thesis "General Bayesian inference schemes in infinite mixture models" by María Dolores Lomelí-García.
Summary updated