In Bayesian inference, we often face a "likelihood-free" problem: we can simulate data from a complex model, but we cannot write down the mathematical formula for the probability of that data. Approximate Bayesian Computation (ABC) solves this by comparing simulated data to observed data. However, as Dennis Prangle notes in Summary Statistics in Approximate Bayesian Computation, ABC suffers from the "curse of dimensionality." If you compare raw, high-dimensional datasets directly, the probability of a simulation landing near the observation becomes vanishingly small.
To make ABC work, researchers must condense raw data into a few highly informative "summary statistics." Historically, these were hand-picked by experts, but a more powerful approach has emerged: using regression to learn the optimal summaries automatically.
The Intuition: The Posterior Mean as the Best Summary
The core breakthrough in this field, pioneered by Paul Fearnhead in Constructing Summary Statistics for Approximate Bayesian Computation: Semi-automatic ABC, is the realization that the posterior mean of a parameter is the theoretically optimal summary statistic.
If your goal is to estimate a parameter $\theta$, the most informative single value you can extract from your data is your best guess of $\theta$ itself. By using the estimated posterior mean as a summary statistic, you project high-dimensional data into a low-dimensional space—specifically, a space with the same dimensionality as your parameters—while preserving the maximum amount of relevant information.
The Semi-Automatic Mechanism
Since we don't know the true posterior mean (that is what we are trying to find), we estimate it using a "semi-automatic" two-step process:
- Pilot Simulations: Generate a large set of simulated parameters and their corresponding data.
- Regression Training: Train a regression model where the simulated data features are the predictors and the parameters are the targets. As Matthew A. Nunes explains in the documentation for abctools: An R Package for Tuning Approximate Bayesian Computation Analyses, this model learns a mapping from the complex data space to the parameter space.
Once trained, this regression model acts as a "distiller." When you perform the actual ABC inference, you pass both your observed data and your new simulations through this model. The resulting predictions serve as the summary statistics. This ensures the statistics are explicitly optimized to distinguish between different parameter values.
From Linear Models to Machine Learning
The "semi-automatic" label originally referred to using linear regression on transformed data features. Paul Fearnhead showed that even simple linear models can capture complex posterior relationships if the data is pre-processed correctly.
However, the framework has expanded to more sophisticated models. Yannik Schälte and the team behind pyABC: Efficient and robust easy-to-use approximate Bayesian computation have integrated machine learning models to automatically extract features from raw data. In more theoretical directions, Dino Sejdinovic introduced DR-ABC: Approximate Bayesian Computation with Kernel-Based Distribution Regression, which uses kernel methods to represent data distributions directly, bypassing the need for manual feature engineering entirely.
Applications in Complex Systems
This regression-based approach is particularly effective for models where the "signal" is buried in noisy, high-dimensional output:
- Biological Imaging: Brenda Vo used semi-automatic ABC in Melanoma Cell Colony Expansion Parameters Revealed by Approximate Bayesian Computation to condense complex image data of cell growth into informative summaries, allowing for the calibration of stochastic motility and adhesion models.
- Environmental Extremes: Xing Ju Lee applied these techniques to spatial extremes models for temperature data, where identifying the most informative features is critical for distinguishing between different max-stable statistical models.
By transforming the problem of "choosing statistics" into a problem of "predicting parameters," regression-based ABC provides a principled, data-driven way to perform inference on the world's most complex generative models.