In many scientific fields, from population genetics to climate science, we have "black-box" simulators that can generate realistic data but lack a tractable likelihood function. To perform Bayesian inference in these settings, researchers traditionally rely on Approximate Bayesian Computation (ABC).
The core of ABC is a simple comparison: if the output of a simulation $x_{sim}$ is "close" to the observed data $x_{obs}$, the parameters $\theta$ that produced $x_{sim}$ are likely. However, comparing high-dimensional raw data directly is computationally impossible—a problem known as the "curse of dimensionality." To solve this, researchers reduce the data into a few summary statistics $S(x)$.
Historically, choosing these statistics was a "black art" requiring deep domain expertise. If the statistics are poorly chosen, they discard critical information, leading to biased or overly broad posterior distributions. Recent research has shifted toward automating this process, transforming feature engineering from a manual hurdle into a learned component of the inference pipeline.
The Optimal Statistic as a Regression Problem
The theoretical foundation for automating this process began with the realization that not all statistics are created equal. In Constructing Summary Statistics for Approximate Bayesian Computation: Semi-automatic ABC, Paul Fearnhead proved that the "optimal" summary statistics—those that minimize the variance of the posterior estimates—are the posterior means of the parameters themselves, $E[\theta|x]$.
Since we don't know the posterior mean (that’s what we’re trying to find), Fearnhead proposed a "semi-automatic" two-step approach:
- Run a pilot set of simulations to generate pairs of $(\theta, x)$.
- Fit a regression model (like linear regression or a neural network) to predict $\theta$ from $x$.
The predicted values from this model then serve as the summary statistics for a second, more rigorous round of ABC. This works because the regression captures the most informative mappings from the data to the parameter space, effectively distilling the high-dimensional observation into a low-dimensional vector of "best guesses."
Inference via Classification
A different perspective emerged from the world of machine learning: if we can't easily define a distance between datasets, perhaps we can learn one. Michael U. Gutmann and Jukka Corander introduced a framework in Likelihood-free inference via classification that reframes inference as a binary classification task.
Instead of manually designing statistics, you train a classifier to distinguish between "real" observed data and "fake" simulated data. If the classifier is unable to tell the difference (i.e., its accuracy is near 50%), then the parameters used to generate the simulated data are likely correct. This approach, further refined in Likelihood-free inference by penalised logistic regression, allows the model to automatically identify which features of the data are most useful for distinguishing between different parameter values, bypassing manual engineering entirely.
Deep Learning and Sequential Estimation
As datasets grew more complex—such as the long time series found in neuroscience—even simple regression became insufficient. Researchers began using deep architectures to extract features from raw data. In Flexible statistical inference for mechanistic models of neural dynamics, Giacomo Bassetto and Jakob H. Macke utilized neural networks to map observed data directly to posterior distributions.
This evolved into Sequential Neural Posterior Estimation (SNPE) and Automatic Posterior Transformation (APT). In Automatic Posterior Transformation for Likelihood-Free Inference, David S. Greenberg showed how neural density estimators (like normalizing flows) can learn the posterior across multiple rounds of simulation. These methods are "summary-statistic-free" in the sense that the network itself learns the optimal representation of the data during training. For example, Mattias Åkesson demonstrated in Convolutional Neural Networks as Summary Statistics for Approximate Bayesian Computation that CNNs can automatically extract local temporal patterns from biological time series that humans would likely miss.
Distance-Based Alternatives
While many focus on learning statistics, another branch of the field seeks to replace them with "full-data" distances. Instead of reducing data to a few numbers, these methods compare the empirical distributions of the datasets directly.
Christopher Drovandi and David T. Frazier have explored using statistical distances like the Wasserstein distance or Maximum Mean Discrepancy (MMD). As discussed in A comparison of likelihood-free methods with and without summary statistics, these distances provide a principled way to compare datasets without the information loss inherent in summarization. Similarly, Bai Jiang proposed using Kullback-Leibler divergence estimators to achieve scalable, automated inference.
Why It Works: The Information Bottleneck
The transition from manual to automated statistics works because it treats the data-reduction step as an optimization problem rather than an intuition problem. By using regression, classification, or neural density estimation, we are essentially training a "bottleneck" that is forced to preserve only the information in the data $x$ that is relevant to the parameters $\theta$.
Whether through the "semi-automatic" regression of Fearnhead or the amortized neural networks in the sbi toolkit, the goal remains the same: to find a low-dimensional representation that makes likelihood-free inference both computationally efficient and statistically rigorous.