Approximate Bayesian Computation via Regression Density Estimation

Yanan Fan, David J. Nott, Scott A. Sisson

Abstract

Approximate Bayesian computation (ABC) methods, which are applicable when the likelihood is difficult or impossible to calculate, are an active topic of current research. Most current ABC algorithms directly approximate the posterior distribution, but an alternative, less common strategy is to approximate the likelihood function. This has several advantages. First, in some problems, it is easier to approximate the likelihood than to approximate the posterior. Second, an approximation to the likelihood allows reference analyses to be constructed based solely on the likelihood. Third, it is straightforward to perform sensitivity analyses for several different choices of prior once an approximation to the likelihood is constructed, which needs to be done only once. The contribution of the present paper is to consider regression density estimation techniques to approximate the likelihood in the ABC setting. Our likelihood approximations build on recently developed marginal adaptation density estimators by extending them for conditional density estimation. Our approach facilitates reference Bayesian inference, as well as frequentist inference. The method is demonstrated via a challenging problem of inference for stereological extremes, where we perform both frequentist and Bayesian inference.

Introduction: The Likelihood Bottleneck

In statistical modeling, the likelihood function is the cornerstone of inference, representing the probability of observing data given a specific set of parameters. However, in complex fields such as genetics, ecology, and materials science, models are often so intricate that the likelihood function becomes analytically intractable. In these "likelihood-free" scenarios, researchers typically rely on Approximate Bayesian Computation (ABC).

Standard ABC methods generally focus on approximating the posterior distribution directly. While effective, this approach tethers the simulation process to a specific prior. If a researcher wishes to test a different prior, the entire computationally expensive simulation suite must often be re-run. Furthermore, standard ABC is inherently Bayesian, making frequentist tools like Maximum Likelihood Estimation (MLE) difficult to implement. This paper proposes a shift in perspective: using regression density estimation to approximate the likelihood function itself, thereby decoupling simulation from inference.

Methodology: Regression Density Estimation

The authors introduce a two-stage framework to "learn" the likelihood from simulated data, utilizing a flexible mixture-model approach.

1. Marginal Estimation via Mixture of Experts

Rather than attempting to model a high-dimensional dataset simultaneously, the method first addresses individual summary statistics. The researchers employ a "mixture of experts" approach, which fits multiple local linear regressions to the data. This allows the model to capture complex, non-linear relationships and heteroscedasticity (changing variances) across the parameter space. These marginal adaptation density estimators provide a robust foundation for the conditional density of the statistics given the parameters.

2. Joint Estimation via Gaussian Copulas

To reconstruct the joint likelihood from these marginals, the authors utilize a Gaussian copula. By transforming the marginal distributions into standard normals, they can model the underlying dependencies using a mixture of multivariate normals. This hybrid approach allows for extreme flexibility in capturing the shape of the likelihood while maintaining computational efficiency, effectively "straightening out" complex data structures for easier navigation.

Case Study: Stereological Extremes

The method was validated using the challenging problem of stereological extremes. The goal was to predict the size of the largest microscopic inclusion (impurity) in a large block of steel based on measurements from a single 2D slice. This is a critical safety metric, as large inclusions can lead to structural failure.

Two models were tested:

  • Spherical Model: Inclusions are assumed to be spheres. In this case, the true likelihood is known, providing a ground-truth benchmark for the regression density estimate.
  • Ellipsoidal Model: A more realistic but complex model where inclusions are ellipsoidal, rendering the likelihood intractable.

Key Results and Efficiency

The regression-based approach demonstrated significant advantages over traditional ABC methods in three key areas:

Computational Efficiency

The authors achieved results comparable to previous "gold standard" methods using only 5,000 simulations. In contrast, previous studies addressing the same dataset required tens of millions of simulations. This represents a massive reduction in the computational overhead required for complex model inference.

Accuracy and Precision

In the spherical model benchmark, the regression density estimate produced a posterior mean for the rate parameter ((\lambda)) of 31.65, remarkably close to the true Markov Chain Monte Carlo (MCMC) result of 30.61. In the ellipsoidal model, the method provided tighter 95% confidence intervals for the scale parameter ((\sigma))—[1.08, 2.96]—compared to the much wider ranges produced by earlier ABC techniques, suggesting superior information extraction.

Prior Sensitivity and Scalability

Because the likelihood is estimated independently of the prior, the authors performed sensitivity analyses for four different priors on the shape parameter ((\xi)) almost instantaneously. Additionally, the method remained stable even when using 115 different summary statistics, showing resilience against the "curse of dimensionality" that often degrades the performance of standard ABC.

Conclusion

By approximating the likelihood rather than the posterior, this regression density estimation approach bridges the gap between Bayesian and frequentist inference in likelihood-free contexts. While the method requires careful selection of summary statistics and initial computational effort to train the mixture models, its ability to facilitate rapid prior sensitivity analysis and provide accurate MLEs makes it a powerful tool for modern statistical modeling. Future work may focus on further automating the selection of summary statistics and improving density estimation at the boundaries of the parameter space.

Summary updated

Abstract

Approximate Bayesian computation (ABC) methods, which are applicable when the likelihood is difficult or impossible to calculate, are an active topic of current research. Most current ABC algorithms directly approximate the posterior distribution, but an alternative, less common strategy is to approximate the likelihood function. This has several advantages. First, in some problems, it is easier to approximate the likelihood than to approximate the posterior. Second, an approximation to the likelihood allows reference analyses to be constructed based solely on the likelihood. Third, it is straightforward to perform sensitivity analyses for several different choices of prior once an approximation to the likelihood is constructed, which needs to be done only once. The contribution of the present paper is to consider regression density estimation techniques to approximate the likelihood in the ABC setting. Our likelihood approximations build on recently developed marginal adaptation density estimators by extending them for conditional density estimation. Our approach facilitates reference Bayesian inference, as well as frequentist inference. The method is demonstrated via a challenging problem of inference for stereological extremes, where we perform both frequentist and Bayesian inference.
BibTeX
@inproceedings{fan2013approximate,
  author = {Yanan Fan and David J. Nott and Scott A. Sisson},
  title = {{Approximate Bayesian Computation via Regression Density Estimation}},
  booktitle = {Stat},
  year = {2013},
  doi = {10.1002/sta4.15},
}
Figure