In many scientific fields—from neuroscience to cosmology—researchers use complex stochastic simulators to model the world. These simulators can generate realistic synthetic data $x$ given a set of parameters $\theta$. However, they are "black boxes": while we can run them forward, we cannot write down the mathematical formula for the probability of the data given the parameters, $p(x|\theta)$.
This is the intractable likelihood problem. Without an explicit likelihood function, traditional Bayesian tools like Markov Chain Monte Carlo (MCMC) cannot be used to find the posterior distribution $p(\theta|x_{obs})$. Recent research has shifted from simple rejection-based methods toward using neural networks to learn these distributions directly.
The Likelihood Gap
Traditional approaches like Approximate Bayesian Computation (ABC) attempt to solve this by running millions of simulations and keeping only those where the synthetic data $x$ is "close" to the observed data $x_{obs}$. As noted by Adeline Samson and colleagues in their work on stochastic neuron models, this often requires carefully chosen "summary statistics" to make the comparison tractable.
Modern Simulation-Based Inference (SBI) replaces this manual comparison with neural density estimators. Instead of throwing away simulations that don't match, we use every simulation to train a model that understands the relationship between $\theta$ and $x$.
Three Paths to the Posterior
Researchers have converged on three primary ways to use neural networks for this task:
1. Learning the Forward Model (NLE)
In Neural Likelihood Estimation (NLE), we train a network to approximate the simulator itself. George Papamakarios and Iain Murray 0001 introduced Sequential Neural Likelihood (SNL), which uses autoregressive flows to learn a surrogate likelihood $\hat{p}(x|\theta)$.
Why it works: By learning the forward process, we can plug this surrogate into standard MCMC samplers. This is often more sample-efficient than learning the posterior directly because the likelihood surface is typically smoother and simpler than the posterior surface.
2. Learning the Inverse Model (NPE)
Neural Posterior Estimation (NPE) aims to learn $p(\theta|x)$ directly. David S. Greenberg and Jakob H. Macke developed Automatic Posterior Transformation (APT) to perform this efficiently.
Why it works: NPE is "amortized." Once the network is trained on a variety of $\theta, x$ pairs, it can produce a posterior for a new observation $x_{obs}$ almost instantaneously. APT specifically allows the model to be trained over multiple rounds, focusing the simulator on the most relevant parts of the parameter space.
3. Inference as Classification (NRE)
Neural Ratio Estimation (NRE) avoids density estimation entirely. Michael U. Gutmann and collaborators proposed LFIRE, which trains a classifier to distinguish between "real" pairs $(\theta, x)$ drawn from the joint distribution and "fake" pairs drawn from the marginals.
Why it works: The optimal classifier for this task naturally learns the likelihood ratio. Since the posterior is proportional to the prior times the likelihood ratio, this ratio is all we need to perform Bayesian inference.
Scaling to High Dimensions
As simulators become more complex, the neural architectures used to model them have evolved.
- Normalizing Flows: Methods like SNL and APT rely on flows to transform simple distributions into complex ones.
- Flow Matching: Jonas Bernhard Wildberger introduced Flow Matching Posterior Estimation (FMPE), which regresses a vector field rather than an invertible mapping. This allows for larger, non-invertible architectures that are faster to train.
- Diffusion Models: Andreas Hellander and Prashant Singh proposed ConDiSim, using conditional diffusion models to capture highly multi-modal posteriors that traditional flows might struggle to represent.
Theoretical Foundations and Robustness
While these methods are empirically powerful, recent work has focused on their statistical validity. David T. Frazier and Christopher Drovandi have provided theoretical proofs showing that NPE and NLE are as statistically sound as classical methods while being far more computationally efficient.
Furthermore, because simulators are often imperfect reflections of reality, researchers like François-Xavier Briol have developed robust bootstrap methods using the Maximum Mean Discrepancy (MMD) to ensure that inference remains accurate even when the model is misspecified.