Abstract
Estimating the parameters of mathematical models is a common problem in almost all branches of science. However, this problem can prove notably difficult when processes and model descriptions become increasingly complex and an explicit likelihood function is not available. With this work, we propose a novel method for globally amortized Bayesian inference based on invertible neural networks which we call BayesFlow. The method uses simulation to learn a global estimator for the probabilistic mapping from observed data to underlying model parameters. A neural network pre-trained in this way can then, without additional training or optimization, infer full posteriors on arbitrary many real datasets involving the same model family. In addition, our method incorporates a summary network trained to embed the observed data into maximally informative summary statistics. Learning summary statistics from data makes the method applicable to modeling scenarios where standard inference techniques with hand-crafted summary statistics fail. We demonstrate the utility of BayesFlow on challenging intractable models from population dynamics, epidemiology, cognitive science and ecology. We argue that BayesFlow provides a general framework for building amortized Bayesian parameter estimation machines for any forward model from which data can be simulated.
Introduction
In many scientific fields—from epidemiology to cognitive science—researchers use stochastic models to describe natural processes. Ideally, we want to perform Bayesian inference to find the hidden parameters ($\theta$) that best explain our observed data ($x$). However, for many complex models, the "likelihood function" (the probability of seeing data given parameters) is mathematically intractable or impossible to compute.
Traditional "likelihood-free" methods, such as Approximate Bayesian Computation (ABC), solve this by running millions of simulations for every single new dataset to see which parameters produce data similar to the observations. This "case-based" approach is incredibly slow and inefficient for large-scale data analysis. BayesFlow addresses this by proposing amortized inference: performing the heavy computational lifting once during a training phase so that subsequent inference is nearly instantaneous.
Methods
The BayesFlow architecture consists of two main deep learning components trained jointly to bypass the likelihood bottleneck:
- Summary Network: This module takes raw, variable-sized data (like time-series or trial-by-trial measurements) and compresses it into a fixed-size vector of "learned summary statistics." Unlike traditional methods that rely on hand-crafted statistics (like means or variances), this network learns the most informative features directly from the data.
- Inference Network (cINN): This is a conditional Invertible Neural Network based on normalizing flows. It learns a bijective (two-way) mapping between the complex parameter space and a simple latent space (usually a standard Gaussian).
During the Forward pass (Training), the network learns to map parameters to the Gaussian space, conditioned on the summary statistics. In the Inverse pass (Inference), the posterior is obtained by sampling from the Gaussian latent space and running those samples backward through the network. Because the network is invertible, it provides a theoretical guarantee for sampling from the true posterior under perfect convergence, without requiring an explicit likelihood function.
Experimental Evaluation
The authors evaluated BayesFlow on several challenging "intractable" models across different scientific domains:
- Toy Models: A Multivariate Normal distribution and a Gaussian Mixture Model to demonstrate accuracy and the ability to capture multimodal posteriors.
- Population Dynamics: The Ricker model, known for chaotic behavior, and the Lotka-Volterra predator-prey model.
- Cognitive Science: The Lévy-Flight Model (LFM), a stochastic differential equation model used to describe perceptual decision-making.
- Epidemiology: The SIR compartmental model describing the spread of infectious diseases.
Key Results
- Inference Speed: For the Lévy-Flight Model, BayesFlow performed inference on 500 datasets in just 3.7 seconds. In contrast, a standard high-quality method (SMC-MMD) would take an estimated 2,700 hours for the same task.
- Accuracy and Calibration: Across all models, BayesFlow achieved high coefficients of determination ($R^2 \approx 0.92 - 0.99$) and low error rates. Using Simulation-Based Calibration (SBC), the authors showed that the approximate posteriors are statistically well-calibrated, meaning the uncertainty estimates are trustworthy.
- Posterior Contraction: The model correctly reflects decreasing uncertainty as more data points ($N$) are provided, with the posterior distribution naturally "contracting" as information increases.
Discussion and Future Work
By training the summary and inference networks together, BayesFlow automatically discovers "maximally informative" statistics that often outperform expert-derived ones. The primary advantage is amortization: while training might take a day, it is done only once. The resulting "inference machine" can be shared, allowing other researchers to analyze their data instantly.
However, the method is dependent on the accuracy of the forward simulator. If the simulator fails to capture real-world nuances, the inference will be biased. Future directions include exploring more complex summary architectures, such as Graph Neural Networks or Attention mechanisms, and developing general-purpose software to make these tools accessible to non-experts.
Summary updated