Approximate Bayesian computation and Bayes linear analysis: Towards high-dimensional ABC

David J. Nott, Yanan Fan, Lucy Marshall, Scott A. Sisson

Abstract

Bayes linear analysis and approximate Bayesian computation (ABC) are techniques commonly used in the Bayesian analysis of complex models. In this article we connect these ideas by demonstrating that regression-adjustment ABC algorithms produce samples for which first and second order moment summaries approximate adjusted expectation and variance for a Bayes linear analysis. This gives regression-adjustment methods a useful interpretation and role in exploratory analysis in high-dimensional problems. As a result, we propose a new method for combining high-dimensional, regression-adjustment ABC with lower-dimensional approaches (such as using MCMC for ABC). This method first obtains a rough estimate of the joint posterior via regression-adjustment ABC, and then estimates each univariate marginal posterior distribution separately in a lower-dimensional analysis. The marginal distributions of the initial estimate are then modified to equal the separately estimated marginals, thereby providing an improved estimate of the joint posterior. We illustrate this method with several examples.

Introduction

Bayesian inference for complex models often encounters a significant hurdle: the likelihood function may be computationally intractable. Approximate Bayesian Computation (ABC) has emerged as a vital tool to bypass this issue by simulating data from the model and comparing it to observed data through a set of summary statistics. However, ABC is notoriously susceptible to the "curse of dimensionality." As the number of parameters and summary statistics increases, the probability of a simulation falling within a reasonable tolerance of the observed data decreases exponentially.

While regression-adjustment techniques—which use local linear models to "correct" simulated samples toward the observed data—mitigate this problem, they often struggle in high-dimensional spaces or when the relationship between parameters and statistics is highly non-linear. This paper bridges the gap between ABC and Bayes linear analysis to provide a more robust framework for high-dimensional inference.

A primary contribution of this work is the formal connection between regression-adjustment ABC and Bayes linear analysis. The authors demonstrate that the widely used regression-adjustment algorithm (specifically the approach by Beaumont et al., 2002) serves as a Monte Carlo approximation of a Bayes linear analysis.

In this context, the regression-adjusted samples produce first and second moments that approximate the adjusted expectation and variance of the posterior. This realization provides a formal justification for using linear adjustments even when the global model is non-linear; it positions the adjustment as a conservative exploratory tool that provides an upper bound on posterior variance, rather than just an ad-hoc correction.

The Marginal Adjustment Strategy

To overcome the limitations of high-dimensional ABC, the authors propose a "marginal adjustment" workflow designed to improve accuracy by decomposing the problem. The strategy follows three distinct steps:

  1. Initial Joint Estimation: Generate a "rough" joint posterior sample using standard regression-adjustment ABC on the full set of parameters and summary statistics. This step captures the multivariate dependence structure (the correlation) between parameters.
  2. Refined Marginal Estimation: For each individual parameter, identify a small, targeted subset of summary statistics that are specifically informative for that parameter. Perform separate, low-dimensional ABC analyses to obtain highly accurate estimates of each univariate marginal posterior.
  3. Quantile Replacement: Use a quantile-replacement method based on order statistics to modify the initial joint sample. By replacing the marginals of the joint sample with the refined univariate estimates, the method preserves the "shape" (copula) of the dependence structure while correcting the location, scale, and distribution of each individual variable.

Empirical Evaluation

The proposed method was evaluated across three diverse scenarios:

Toy Gaussian Mixture Model

Using a $p$-dimensional model with $2^p$ mixture components, the authors compared the marginal adjustment method against standard rejection and linear-adjustment ABC. As dimensionality increased, the marginal adjustment method maintained a consistently low Kullback-Leibler (KL) divergence from the true posterior, whereas the errors of standard methods exploded.

Heather Incidence Data

In a spatial analysis of heather presence modeled via Gaussian random fields, the joint regression approach initially produced marginals that were incorrectly centered. The marginal adjustment strategy successfully shifted these estimates to their correct locations, significantly improving the accuracy of bivariate density estimates.

Hydrological Modeling (AWBM)

The Australian Water Balance Model (AWBM) presented a high-dimensional challenge involving 10 parameters and complex rainfall-runoff time series. The unadjusted joint regression produced a posterior estimate that was deemed "highly inappropriate" for the observed streamflow. In contrast, the marginally adjusted posterior provided a significantly better fit to the data, yielding a much lower median absolute error.

Discussion and Limitations

The "divide and conquer" approach of marginal adjustment proves robust because estimating 1D marginals is significantly easier than estimating a 10D joint distribution. However, the method is not without limitations. It relies heavily on the initial joint sample to capture the correct dependence structure. If the initial high-dimensional ABC is so poor that it fails to capture the underlying correlations, marginal adjustment cannot correct the "shape" of the joint posterior. Additionally, researchers must be cautious of parameter constraints (e.g., $\theta_1 + \theta_2 < 1$), as adjusted marginals might theoretically fall outside valid joint regions, though reparameterization often mitigates this risk.

By framing regression-ABC within the context of Bayes linear analysis, this work provides both a theoretical foundation and a practical pipeline for extending the reach of Bayesian inference into higher-dimensional regimes.

Summary updated

Abstract

Bayes linear analysis and approximate Bayesian computation (ABC) are techniques commonly used in the Bayesian analysis of complex models. In this article we connect these ideas by demonstrating that regression-adjustment ABC algorithms produce samples for which first and second order moment summaries approximate adjusted expectation and variance for a Bayes linear analysis. This gives regression-adjustment methods a useful interpretation and role in exploratory analysis in high-dimensional problems. As a result, we propose a new method for combining high-dimensional, regression-adjustment ABC with lower-dimensional approaches (such as using MCMC for ABC). This method first obtains a rough estimate of the joint posterior via regression-adjustment ABC, and then estimates each univariate marginal posterior distribution separately in a lower-dimensional analysis. The marginal distributions of the initial estimate are then modified to equal the separately estimated marginals, thereby providing an improved estimate of the joint posterior. We illustrate this method with several examples. Supplementary materials for this article are available online.
BibTeX
@article{nott2012approximate,
  author = {David J. Nott and Yanan Fan and Lucy Marshall and Scott A. Sisson},
  title = {{Approximate Bayesian computation and Bayes linear analysis: Towards high-dimensional ABC}},
  journal = {Journal of Computational and Graphical Statistics},
  year = {2012},
  doi = {10.1080/10618600.2012.751874},
}
Figure