Bayesian Inference on Mixtures of Distributions

Kate Lee, Jean‐Michel Marin, Inria Saclay, Kerrie MengersenarXiv 2008 · DOI · arXiv

Abstract

This survey covers state-of-the-art Bayesian techniques for the estimation of mixtures. It complements the earlier Marin, Mengersen and Robert (2005) by studying new types of distributions, the multinomial, latent class and t distributions. It also exhibits closed form solutions for Bayesian inference in some discrete setups. Lastly, it sheds a new light on the computation of Bayes factors via the approximation of Chib (1995).

Introduction: The Challenge of $J^n$ Complexity

Mixture models are essential for representing complex, multimodal data by assuming observations arise from a weighted combination of several simpler component distributions. However, they present significant computational hurdles. The primary challenge is the "likelihood explosion": calculating the likelihood for $n$ observations across $J$ components requires evaluating $J^n$ possible group assignments. Furthermore, the "label switching" phenomenon—where components are mathematically interchangeable—creates a posterior distribution with $J!$ identical modes, complicating standard Markov Chain Monte Carlo (MCMC) convergence.

Methodology

The paper explores three primary Bayesian approaches to address these challenges:

1. Exact Posterior Computation

For discrete distributions within the natural exponential family (e.g., Poisson or Multinomial), the authors demonstrate that exact inference is possible without relying on MCMC. By employing a recursive "book-keeping" algorithm, the $J^n$ terms are collapsed into a manageable set of sufficient statistics. This allows for precise results in small-to-medium datasets.

Example of a simulated discrete distribution used for testing exact inference.

2. Data Augmentation and Gibbs Sampling

This standard approach treats unknown group assignments as missing data. The algorithm alternates between assigning data points to components and updating component parameters. While robust, it is susceptible to the label-switching problem.

3. Symmetrized Chib’s Approximation

To determine the optimal number of components ($J$), Bayes factors are typically used. The popular Chib (1995) method often fails in mixture contexts because MCMC chains become trapped in a single symmetric mode. The authors propose a symmetrized version that averages over all permutations, significantly improving accuracy in model selection.

Empirical Validation and Results

The methods were validated using diverse datasets, including galaxy radial speeds, aerosol diameters, and sociological survey data.

A bimodal mixture distribution from simulated Poisson data, illustrating the complexity of component separation.

Key Findings

  • Model Selection: On the benchmark Galaxy dataset, the symmetrized Chib’s approximation identified a 5-component model as the most probable ($log$-marginal likelihood of -101.93), correcting previous underestimations caused by the label-switching trap.
  • Computational Boundaries: While exact recursive algorithms are powerful, they are memory-intensive. For a Poisson mixture with $J=4$ and $n=10$, over 120,000 sufficient statistics must be tracked, highlighting the eventual necessity of MCMC for larger scales.
  • Robustness of $t$-Distributions: In modeling aerosol data, mixtures of Student’s $t$-distributions proved more flexible than Normal mixtures, better capturing the "tails" and outliers of real-world distributions.

Discussion and Future Directions

While the paper advances Bayes factor estimation and exact inference, several challenges remain. Determining the number of components ($J$) is still computationally taxing. Additionally, in Student’s $t$ mixtures, the degrees of freedom parameter often has a flat posterior, indicating limited data informativeness regarding tail heaviness. Future work involves extending exact "book-keeping" efficiencies to continuous distributions and scaling these methods for larger datasets.

Summary updated

Abstract

This survey covers state-of-the-art Bayesian techniques for the estimation of mixtures. It complements the earlier Marin, Mengersen and Robert (2005) by studying new types of distributions, the multinomial, latent class and t distributions. It also exhibits closed form solutions for Bayesian inference in some discrete setups. Lastly, it sheds a new light on the computation of Bayes factors via the approximation of Chib (1995).
BibTeX
@misc{lee2008bayesian,
  author = {Kate Lee and Jean‐Michel Marin and Inria Saclay and Kerrie Mengersen},
  title = {{Bayesian Inference on Mixtures of Distributions}},
  year = {2008},
  eprint = {0804.2413},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/0804.2413},
}
Figure