Abstract
For stochastic models with intractable likelihood functions, approximate Bayesian computation offers a way of approximating the true posterior through repeated comparisons of observations with simulated model outputs in terms of a small set of summary statistics. These statistics need to retain the information that is relevant for constraining the parameters but cancel out the noise. They can thus be seen as thermodynamic state variables, for general stochastic models. For many scientific applications, we need strictly more summary statistics than model parameters to reach a satisfactory approximation of the posterior. Therefore, we propose to use the inner dimension of deep neural network based Autoencoders as summary statistics. To create an incentive for the encoder to encode all the parameter-related information but not the noise, we give the decoder access to explicit or implicit information on the noise that has been used to generate the training data. We validate the approach empirically on two types of stochastic models.
Introduction
In fields such as ecology, physics, and climate science, stochastic models are essential for capturing the inherent randomness of complex systems. However, calibrating these models to real-world data often involves Bayesian inference with "intractable likelihoods," where calculating the probability of specific observations is computationally prohibitive.
Approximate Bayesian Computation (ABC) offers a solution by comparing observed data with simulated outputs using a set of summary statistics. The effectiveness of ABC depends heavily on these statistics; poor choices lead to biased results or extreme computational inefficiency. While regression-based methods are common, they often fail to capture the global structure or "phases" of complex distributions.
Methodology: Noise-Conditional Autoencoders
The authors propose using Autoencoders (AEs) to automatically learn optimal summary statistics. By treating these statistics as thermodynamic state variables, the model can describe the "state" of a stochastic system without needing to track every random fluctuation. To ensure the encoder ignores noise and focuses on underlying parameters, two novel architectures are introduced:
Explicit Noise Conditional Autoencoder (ENCA)
In the ENCA architecture, the decoder is provided with the exact "bare noise" ($\epsilon$) used to generate the training simulations. This creates an information-theoretic incentive for the encoder to ignore the noise, as that information is already available to the decoder, forcing the bottleneck to capture only parameter-related features.
Implicit Noise Conditional Autoencoder (INCA)
For scenarios where noise cannot be explicitly isolated, the INCA architecture uses "replicas"—multiple simulations generated from the same parameters. The decoder analyzes these replicas to distinguish consistent signals (parameters) from random jitter (noise).
Experimental Validation
The approach was validated on two challenging stochastic models:
- Bi-modal Map: A system that settles into one of two stable "attractors," testing the ability to distinguish between different system phases.
- Complex Noise Map: A model with both additive and multiplicative noise, lacking a simple set of sufficient statistics.
Key Results and Insights
The researchers found that for complex models, a "More is More" principle applies: using more summary statistics than model parameters often yields better posterior approximations. These extra statistics act as "order parameters" describing global behavior.
Empirical results showed that AE-learned statistics produced posterior distributions significantly closer to the ground truth than standard regression methods. Notably, the AEs successfully identified "order parameters" that could distinguish between system attractors, a task where traditional statistics often fail. However, a "sweet spot" exists; adding too many statistics can eventually lead to overfitting the noise.
Conclusion and Future Work
While noise-conditional autoencoders significantly improve Bayesian inference for complex stochastic models, they come with higher initial computational costs for training. Future research will focus on scaling this method to high-dimensional spatial models and using symbolic regression to translate learned latent features into interpretable mathematical formulas.
Summary updated