In fields ranging from systems biology to cosmology, scientists use complex, stochastic simulators to model real-world phenomena. These simulators can generate realistic data $x$ given a set of parameters $\theta$. However, evaluating the likelihood function $p(x|\theta)$—the probability of observing the data given the parameters—is often computationally intractable because it requires integrating over all possible internal states of the simulator.
To perform Bayesian inference without an explicit likelihood function, researchers turn to Simulation-Based Inference (SBI) or Approximate Bayesian Computation (ABC). The core challenge of these methods lies in how they handle high-dimensional data.
The Feature Engineering Bottleneck
Traditional ABC algorithms operate by sampling parameters from a prior, running the simulator, and comparing the simulated data to the observed data. If the simulated data is "close enough" to the observed data, the parameters are accepted.
However, comparing raw, high-dimensional datasets (such as long time-series or images) directly is plagued by the curse of dimensionality. In high-dimensional spaces, almost all data points are far apart, forcing algorithms to use a large acceptance threshold ($\epsilon$), which introduces severe bias into the estimated posterior distribution.
To bypass this, traditional frameworks rely on hand-crafted summary statistics $S(x)$ to compress high-dimensional data into a low-dimensional vector. The comparison is then performed on these summaries: $\|S(x_{\text{sim}}) - S(x_{\text{obs}})\| \le \epsilon$.
As noted by Sejdinovic et al. and Papamakarios et al., this manual feature engineering creates a critical bottleneck:
- Information Loss: Unless the summary statistics are mathematically sufficient, they discard information. If the chosen features fail to capture critical dynamics, the resulting posterior will be overly broad or biased (Lueckmann et al.).
- Subjectivity and Domain Dependence: Selecting these statistics requires extensive trial-and-error and deep domain expertise, limiting the scalability and objectivity of the inference pipeline across different scientific fields (Drovandi et al.; Gutmann).
To solve this, modern research focuses on automatically learning optimal summary statistics directly from data distributions, ensuring that no informative signals are lost.
Strategy 1: Learning Sufficiency via Mutual Information
How do we force a neural network to compress data $x$ into a low-dimensional summary $S(x)$ without losing the information necessary to infer $\theta$?
The mathematical answer lies in Information Theory. We want to maximize the mutual information $I(\theta; S(x))$ between the parameters and the learned summary statistics. By making $S(x)$ maximally predictive of $\theta$, the network naturally learns an approximation of a sufficient statistic.
High-Dimensional Data (x)
│
▼
┌───────────────────────┐
│ Neural Network │ ◄─── Optimize to maximize Mutual Information I(θ; S(x))
└───────────────────────┘
│
▼
Low-Dimensional Summary S(x) ───► Retains all information relevant to θ
Chen et al. formalized this by framing summary statistic identification as an explicit mutual information maximization problem. By training neural networks with an information-theoretic objective, the network automatically discovers compact representations that preserve parameter-relevant variance while discarding simulator noise.
Strategy 2: End-to-End Amortized Embedding Networks
Rather than separating the task into two steps (first learning statistics, then running inference), another prominent approach embeds the summary statistic learning directly into an end-to-end generative framework. This is known as Amortized Bayesian Inference.
In frameworks like Sequential Neural Posterior Estimation (SNPE) and Automatic Posterior Transformation (APT), introduced by Lueckmann et al. and Greenberg et al., a neural network is trained to directly output the parameters of a conditional distribution $p(\theta|x)$.
Similarly, the BayesFlow architecture (Radev et al.; Schmitt et al.) pairs a summary network with an Invertible Neural Network (INN). The summary network compresses the raw data into a fixed-size vector, which the INN then uses to map the complex posterior distribution to a simple Gaussian standard normal.
Tailoring Networks to Data Symmetries
To make this learning sample-efficient, the architecture of the summary network must match the structural properties (inductive biases) of the simulated data:
- Temporal Data: For biological or physical time-series, Åkesson et al. demonstrated that Convolutional Neural Networks (CNNs) can automatically extract local temporal patterns that human experts routinely miss.
- Permutation Invariance and Markov Chains: The authors of Partially Exchangeable Networks showed that if the data is generated by a Markov process, we can bake this probabilistic symmetry directly into the network architecture. This allows the model to learn exact summary statistics using up to 100 times less training data than standard architectures.
Strategy 3: Bypassing Summarization via Statistical Distances
An alternative paradigm avoids explicit dimensionality reduction altogether by changing how datasets are compared. Instead of reducing a dataset to a vector of summary statistics, why not treat the entire simulated output as an empirical probability distribution and measure the distance between the simulated and observed distributions directly?
Manual Approach: Raw Data ──► Hand-crafted Summaries ──► Euclidean Distance
Distance Approach: Raw Data ────────────────────────────► Non-parametric Distance (MMD/Wasserstein)
As reviewed by Drovandi et al., methods utilizing non-parametric statistical distances—such as the Wasserstein distance, Cramer-von Mises distance, or Maximum Mean Discrepancy (MMD)—provide an automated alternative to feature engineering.
For instance, Jiang introduced an ABC framework utilizing a linearithmic Kullback-Leibler (KL) divergence estimator as the data discrepancy metric, bypassing summary statistics entirely while maintaining computational efficiency. Similarly, Dellaporta et al. leveraged MMD within a posterior bootstrap framework to achieve inference that is automatically robust to model misspecification without requiring manual feature tuning.
Summary
The transition from manual feature engineering to automated representation learning has transformed simulation-based inference. Whether by optimizing neural networks to maximize mutual information, embedding specialized summary architectures into amortized normalizing flows, or utilizing full-data statistical distances, modern likelihood-free inference eliminates human bias and preserves the critical information hidden within complex scientific data.
Key Foundational Work
K2-ABC: Approximate Bayesian Computation with Kernel Embeddings (2016) — contributed 1 concept elements to this direction.
Fast ε-free Inference of Simulation Models with Bayesian Conditional Density Estimation (2016) — contributed 1 concept elements to this direction.
DR-ABC: Approximate Bayesian Computation with Kernel-Based Distribution Regression (2016) — contributed 1 concept elements to this direction.