Abstract
This survey covers state-of-the-art Bayesian techniques for the estimation of mixtures. It complements the earlier Marin, Mengersen and Robert (2005) by studying new types of distributions, the multinomial, latent class and t distributions. It also exhibits closed form solutions for Bayesian inference in some discrete setups. Lastly, it sheds a new light on the computation of Bayes factors via the approximation of Chib (1995).
Introduction: The Challenge of $J^n$ Complexity
Mixture models are essential for representing complex, multimodal data by assuming observations arise from a weighted combination of several simpler component distributions. However, they present significant computational hurdles. The primary challenge is the "likelihood explosion": calculating the likelihood for $n$ observations across $J$ components requires evaluating $J^n$ possible group assignments. Furthermore, the "label switching" phenomenon—where components are mathematically interchangeable—creates a posterior distribution with $J!$ identical modes, complicating standard Markov Chain Monte Carlo (MCMC) convergence.
Methodology
The paper explores three primary Bayesian approaches to address these challenges:
1. Exact Posterior Computation
For discrete distributions within the natural exponential family (e.g., Poisson or Multinomial), the authors demonstrate that exact inference is possible without relying on MCMC. By employing a recursive "book-keeping" algorithm, the $J^n$ terms are collapsed into a manageable set of sufficient statistics. This allows for precise results in small-to-medium datasets.
2. Data Augmentation and Gibbs Sampling
This standard approach treats unknown group assignments as missing data. The algorithm alternates between assigning data points to components and updating component parameters. While robust, it is susceptible to the label-switching problem.
3. Symmetrized Chib’s Approximation
To determine the optimal number of components ($J$), Bayes factors are typically used. The popular Chib (1995) method often fails in mixture contexts because MCMC chains become trapped in a single symmetric mode. The authors propose a symmetrized version that averages over all permutations, significantly improving accuracy in model selection.
Empirical Validation and Results
The methods were validated using diverse datasets, including galaxy radial speeds, aerosol diameters, and sociological survey data.
Key Findings
- Model Selection: On the benchmark Galaxy dataset, the symmetrized Chib’s approximation identified a 5-component model as the most probable ($log$-marginal likelihood of -101.93), correcting previous underestimations caused by the label-switching trap.
- Computational Boundaries: While exact recursive algorithms are powerful, they are memory-intensive. For a Poisson mixture with $J=4$ and $n=10$, over 120,000 sufficient statistics must be tracked, highlighting the eventual necessity of MCMC for larger scales.
- Robustness of $t$-Distributions: In modeling aerosol data, mixtures of Student’s $t$-distributions proved more flexible than Normal mixtures, better capturing the "tails" and outliers of real-world distributions.
Discussion and Future Directions
While the paper advances Bayes factor estimation and exact inference, several challenges remain. Determining the number of components ($J$) is still computationally taxing. Additionally, in Student’s $t$ mixtures, the degrees of freedom parameter often has a flat posterior, indicating limited data informativeness regarding tail heaviness. Future work involves extending exact "book-keeping" efficiencies to continuous distributions and scaling these methods for larger datasets.
Summary updated