Abstract: In this paper, we show how a complete and exact Bayesian analysis of a parametric mixture model is possible in some cases when components of the mixture are taken from exponential families and when conjugate priors are used. This restricted set-up allows us to show the relevance of the Bayesian approach as well as to exhibit the limitations of a complete analysis, namely that it is impossible to conduct this analysis when the sample size is too large, when the data are not from an exponential family, or when priors that are more complex than conjugate priors are used.
Introduction
Mixture models, such as those combining multiple Poisson or Normal distributions, are fundamental tools in statistics for modeling heterogeneous populations. However, they present significant analytical challenges due to "missing data"—the unknown component assignments for each observation. While standard practice relies on approximate methods like Markov Chain Monte Carlo (MCMC) or the Expectation-Maximization (EM) algorithm, this paper explores the feasibility of a complete and exact Bayesian analysis. By defining the boundaries of where exact calculation is possible, the authors provide a "gold standard" for evaluating approximate methods.
Methodology
The authors focus on a specific mathematical framework: mixtures where components are drawn from natural exponential families and priors are locally conjugate. This setup ensures that the posterior distribution is itself a finite mixture of conjugate distributions.
The exact posterior is derived by summing over all $k^n$ possible assignments of $n$ data points to $k$ components. To manage this combinatorial explosion, the authors utilize a recursive algorithm that groups assignments by their sufficient statistics. For discrete distributions like the Poisson, many assignments yield identical sufficient statistics (e.g., the same group sizes and sums), allowing for significant compression of the posterior terms.
Computational Complexity and the "Wall"
The primary challenge of exact analysis is the rapid growth of unique terms in the posterior mixture. Even with the compression afforded by sufficient statistics, the number of terms eventually exhausts computational resources.
As shown in the table above, for a Poisson mixture with $n=10$ and $k=2$, the $2^{10} = 1024$ possible assignments collapse into as few as 11 unique terms. However, as $n$ and $k$ increase, the complexity scales dramatically. For $n=40$ and $k=2$, there are over 2,600 terms; for $k=4$, the number of terms reaches hundreds of thousands, illustrating the "computational wall" that limits this approach to relatively small sample sizes.
Empirical Evaluation
The approach was tested on both simulated and real-world data:
- Simulated Poisson and Multinomial Mixtures: Used to map the limits of the recursive algorithm across varying $n$ and $k$.
- The "Minnow" Dataset: A real-world application involving 50 observations of fish lengths, modeled as a two-component Poisson mixture.
The results highlight that while discrete models (Poisson, Multinomial) benefit from sufficient statistic grouping, continuous models like Normal mixtures do not. In Normal mixtures, sufficient statistics (means and variances) rarely coincide perfectly, causing the number of terms to scale with the Bell number, making exact analysis practically impossible for all but the smallest samples.
Conclusion and Limitations
This work serves as a formal exercise in understanding the mathematical structure of mixture models. While the exact approach is limited by scalability—becoming a "small $n$" technique—it provides a vital baseline. It allows researchers to verify the accuracy of simulation-based approximations in controlled settings. Future research may explore pruning "low-weight" terms from the exact posterior to extend the reach of this method while maintaining high precision.
Summary updated