General Bayesian inference schemes in infinite mixture models

María LomelíarXiv 2017 · DOI · arXiv

Abstract

Bayesian statistical models allow us to formalise our knowledge about the world and reason about our uncertainty, but there is a need for better procedures to accurately encode its complexity. One way to do so is through compositional models, which are formed by combining blocks consisting of simpler models. This thesis expands the choice of Bayesian nonparametric (BNP) blocks for constructing tractable compositional models. While most BNP models use a Dirichlet Process or a Pitman-Yor process, this work shows how to overcome intractabilities to obtain analogous representations for the broader class of Poisson-Kingman priors. The main contribution is a variety of exact inference schemes—Markov chain Monte Carlo and Sequential Monte Carlo methods—that target the true posterior, providing general-purpose tools for probabilistic programming.

Introduction

Bayesian Nonparametrics (BNP) is a powerful framework for clustering where the number of clusters ($k$) is not fixed but grows with the data. While the Dirichlet Process (DP) and Pitman–Yor (PY) process are widely used due to their tractable representations, they represent only a small subset of the potential models. The broader family of Poisson–Kingman (PK) priors offers significantly more flexibility in capturing diverse clustering behaviors, yet they have remained underutilized due to computational intractability.

This research unlocks the PK family by providing novel, exact inference algorithms that make these flexible models computationally practical for complex, real-world datasets.

The Challenge: Beyond the Dirichlet Process

In standard mixture modeling, pre-specifying the number of components often leads to overfitting or underfitting. BNP models address this by assuming an infinite number of potential components. However, inference in the general PK family is notoriously difficult because their probability densities often involve intractable integrals. Previous research often relied on crude approximations or restricted themselves to the simpler DP/PY cases. This work develops exact inference schemes that target the true posterior without systematic bias, turning numerical integration challenges into standard sampling problems.

Methodology

Three main computational contributions are introduced to facilitate inference in PK models:

  1. The Marginal Sampler: This approach integrates out the infinite-dimensional component. For $\sigma$-stable PK priors, a data augmentation trick based on the Zolotarev integral representation is employed to handle intractable densities.
  2. The Hybrid Sampler: This novel MCMC scheme combines the strengths of marginal and conditional samplers. It represents the model using the weights of occupied clusters and a single "surplus mass" term for all unoccupied clusters. This approach is memory-efficient and avoids the truncation errors typical of slice samplers.
  3. Sequential Monte Carlo (SMC): SMC schemes are developed for sequential inference and estimating the marginal likelihood. This enables formal Bayesian hypothesis testing to compare infinite models against finite alternatives.

Experimental Evaluation

The proposed methods were validated on two primary datasets:

  • Galaxy Dataset: A benchmark of 82 galaxy velocities used to test clustering efficiency.
  • Oil Profiles Dataset: A high-dimensional dataset of 120 triacylglyceride profiles (reduced via PCA). The goal was to cluster oil types (olive, vegetable, blends) without prior knowledge of the number of subtypes.

Key Results

  • Efficiency: The Hybrid Sampler significantly outperformed existing conditional slice samplers. On the Galaxy dataset, it achieved a much higher Effective Sample Size (ESS) per second, demonstrating superior exploration of the posterior landscape.
  • Stability: For specific parameter ranges (e.g., $\sigma=0.5$), the hybrid sampler proved more numerically stable than previous methods.
  • Real-World Utility: On the oil dataset, PK-based mixture models successfully recovered meaningful clusters and provided better predictive probabilities than the standard DP, proving that the added flexibility of PK priors translates to better model performance.

Discussion and Future Work

The "surplus mass" concept in the hybrid sampler allows for an exact representation of an infinite-dimensional model using finite memory. By providing a general-purpose framework for PK priors, this work moves BNP closer to the "probabilistic programming" ideal.

However, challenges remain regarding scalability to datasets with millions of points. Future work may involve extending mathematical identities to the full range of PK priors and incorporating "split-merge" proposals to further improve MCMC mixing speeds in complex clustering configurations.

This summary is based on the PhD thesis "General Bayesian inference schemes in infinite mixture models" by María Dolores Lomelí-García.

Summary updated

Abstract

Bayesian statistical models allow us to formalise our knowledge about the world and reason about our uncertainty, but there is a need for better procedures to accurately encode its complexity. One way to do so is through compositional models, which are formed by combining blocks consisting of simpler models. One can increase the complexity of the compositional model by either stacking more blocks or by using a not-so-simple model as a building block. This thesis is an example of the latter. One first aim is to expand the choice of Bayesian nonparametric (BNP) blocks for constructing tractable compositional models. So far, most of the models that have a Bayesian nonparametric component use a Dirichlet Process or a Pitman-Yor process because of the availability of tractable and compact representations. This thesis shows how to overcome certain intractabilities in order to obtain analogous compact representations for the class of Poisson-Kingman priors which includes the Dirichlet and Pitman-Yor processes. A major impediment to the widespread use of Bayesian nonparametric building blocks is that inference is often costly, intractable or difficult to carry out. This is an active research area since dealing with the model's infinite dimensional component forbids the direct use of standard simulation-based methods. The main contribution of this thesis is a variety of inference schemes that tackle this problem: Markov chain Monte Carlo and Sequential Monte Carlo methods, which are exact inference schemes since they target the true posterior. The contributions of this thesis, in a larger context, provide general purpose exact inference schemes in the flavour or probabilistic programming: the user is able to choose from a variety of models, focusing only on the modelling part. Indeed, if the wide enough class of Poisson-Kingman priors is used as one of our blocks, this objective is achieved.
BibTeX
@misc{lomeli2017general,
  author = {María Lomelí},
  title = {{General Bayesian inference schemes in infinite mixture models}},
  year = {2017},
  eprint = {1702.08781},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1702.08781},
}
Figure