Exact Bayesian Analysis of Mixtures

Christian P. Robert, Kerrie MengersenWiley 2011 · DOI · arXiv · OpenAlex

Abstract: In this paper, we show how a complete and exact Bayesian analysis of a parametric mixture model is possible in some cases when components of the mixture are taken from exponential families and when conjugate priors are used. This restricted set-up allows us to show the relevance of the Bayesian approach as well as to exhibit the limitations of a complete analysis, namely that it is impossible to conduct this analysis when the sample size is too large, when the data are not from an exponential family, or when priors that are more complex than conjugate priors are used.

Introduction

Mixture models, such as those combining multiple Poisson or Normal distributions, are fundamental tools in statistics for modeling heterogeneous populations. However, they present significant analytical challenges due to "missing data"—the unknown component assignments for each observation. While standard practice relies on approximate methods like Markov Chain Monte Carlo (MCMC) or the Expectation-Maximization (EM) algorithm, this paper explores the feasibility of a complete and exact Bayesian analysis. By defining the boundaries of where exact calculation is possible, the authors provide a "gold standard" for evaluating approximate methods.

Methodology

The authors focus on a specific mathematical framework: mixtures where components are drawn from natural exponential families and priors are locally conjugate. This setup ensures that the posterior distribution is itself a finite mixture of conjugate distributions.

The exact posterior is derived by summing over all $k^n$ possible assignments of $n$ data points to $k$ components. To manage this combinatorial explosion, the authors utilize a recursive algorithm that groups assignments by their sufficient statistics. For discrete distributions like the Poisson, many assignments yield identical sufficient statistics (e.g., the same group sizes and sums), allowing for significant compression of the posterior terms.

Computational Complexity and the "Wall"

The primary challenge of exact analysis is the rapid growth of unique terms in the posterior mixture. Even with the compression afforded by sufficient statistics, the number of terms eventually exhausts computational resources.

Table 1: Number of unique terms in the posterior for a Poisson mixture across different sample sizes (n), means (λ), and component counts (k).

As shown in the table above, for a Poisson mixture with $n=10$ and $k=2$, the $2^{10} = 1024$ possible assignments collapse into as few as 11 unique terms. However, as $n$ and $k$ increase, the complexity scales dramatically. For $n=40$ and $k=2$, there are over 2,600 terms; for $k=4$, the number of terms reaches hundreds of thousands, illustrating the "computational wall" that limits this approach to relatively small sample sizes.

Empirical Evaluation

The approach was tested on both simulated and real-world data:

  • Simulated Poisson and Multinomial Mixtures: Used to map the limits of the recursive algorithm across varying $n$ and $k$.
  • The "Minnow" Dataset: A real-world application involving 50 observations of fish lengths, modeled as a two-component Poisson mixture.

The results highlight that while discrete models (Poisson, Multinomial) benefit from sufficient statistic grouping, continuous models like Normal mixtures do not. In Normal mixtures, sufficient statistics (means and variances) rarely coincide perfectly, causing the number of terms to scale with the Bell number, making exact analysis practically impossible for all but the smallest samples.

Conclusion and Limitations

This work serves as a formal exercise in understanding the mathematical structure of mixture models. While the exact approach is limited by scalability—becoming a "small $n$" technique—it provides a vital baseline. It allows researchers to verify the accuracy of simulation-based approximations in controlled settings. Future research may explore pruning "low-weight" terms from the exact posterior to extend the reach of this method while maintaining high precision.

Summary updated

Abstract

In this paper, we show how a complete and exact Bayesian analysis of a\nparametric mixture model is possible in some cases when components of the\nmixture are taken from exponential families and when conjugate priors are used.\nThis restricted set-up allows us to show the relevance of the Bayesian approach\nas well as to exhibit the limitations of a complete analysis, namely that it is\nimpossible to conduct this analysis when the sample size is too large, when the\ndata are not from an exponential family, or when priors that are more complex\nthan conjugate priors are used.\n
BibTeX
@inproceedings{robert2011exact,
  author = {Christian P. Robert and Kerrie Mengersen},
  title = {{Exact Bayesian Analysis of Mixtures}},
  booktitle = {Wiley},
  year = {2011},
  doi = {10.1002/9781119995678.ch11},
}

Loading related content...

Figure