Introduction
In many scientific fields, such as epidemiology or cosmology, researchers rely on "simulator-based" or implicit models. While these simulators can generate data, they lack a tractable mathematical formula for the probability of seeing a specific outcome (the likelihood). To perform Bayesian inference on these models, researchers turn to Likelihood-Free Inference (LFI).
Because raw data—like high-resolution images or long time series—is often too complex for LFI algorithms, it is typically compressed into "summary statistics." Traditionally, these statistics are handcrafted by domain experts. However, if the chosen statistics are poor, the resulting inference can be biased or overconfident. This paper addresses the "black art" of feature engineering by automating the discovery of sufficient statistics: the minimal set of numbers containing all the information needed to estimate a model's parameters. By framing this search as a mutual information maximization problem, the authors significantly improve LFI accuracy without relying on expert-designed features.
Methods: Automating Feature Discovery
The authors propose Neural Approximate Sufficient Statistics. The core mathematical insight is that a statistic is "sufficient" if and only if it maximizes the Mutual Information (MI) between the model parameters ($\theta$) and the compressed data representation ($s(x)$).
To implement this, the authors utilize:
- Infomax Learning: A neural network acts as a data compressor. Instead of training it to predict parameters directly (a common but suboptimal approach), it is trained to maximize MI using two estimators:
- Jensen-Shannon Divergence (JSD): A robust, contrastive-learning-based approach.
- Distance Correlation (DC): A faster, ratio-free estimator that avoids the need for an extra "critic" network.
- Dynamic Learning: Statistics are not just learned once. The authors propose a multi-round strategy where both the statistics and the posterior distribution are iteratively refined. As the algorithm narrows down where the true parameters lie, it simulates more relevant data, helping the network find even better statistics.
- New Algorithms: These neural statistics are integrated into popular LFI frameworks, resulting in SMC-ABC+ (Sequential Monte Carlo Approximate Bayesian Computation) and SNL+ (Sequential Neural Likelihood).
Datasets and Key Results
The approach was evaluated on three distinct data types to demonstrate its versatility:
- Ising Model: A classic physics model representing atomic spins on a lattice (spatial/graph data).
- Gaussian Copula Model: A statistical model testing how well the method handles i.i.d. data with complex dependencies.
- Ornstein-Uhlenbeck (OU) Process: A stochastic differential equation common in finance and physics (time-series data).
Performance Boost: Across all three models, the enhanced "plus" versions (SMC-ABC+ and SNL+) significantly outperformed standard versions relying on raw data or basic statistics.
Matching Experts: The neural statistics performed comparably to—and sometimes better than—handcrafted statistics. In the OU process, the network automatically discovered features as useful as the mean, standard error, and autocorrelation.
Efficiency: While the JSD estimator provided the highest accuracy, the Distance Correlation (DC) estimator was roughly 15 times faster to train, making it highly viable for large-scale, compute-constrained problems.
Key Insights and Future Directions
- Global vs. Local Sufficiency: Previous automatic methods often focused on being accurate near a specific parameter guess. This new method targets global sufficiency, ensuring statistics remain informative across the entire parameter space.
- Moving Beyond the Mean: A common LFI shortcut uses the "posterior mean" as a statistic. The authors mathematically prove that this is generally insufficient because it ignores distribution shape and uncertainty. Their MI-based approach captures the full information profile.
- The $d=2K$ Heuristic: Determining the optimal number of summary statistics remains an open question. However, the authors suggest a practical heuristic: use twice as many statistics as there are parameters ($K$). This provides enough dimensionality for the network to capture complex relationships while remaining compact.
While the method is highly effective, the authors note that contrastive learning can be less stable than direct likelihood estimation in certain high-dimensional spaces (like the Gaussian Copula task). Future work may explore task-specific architectures—such as Transformers for sequences or Graph Neural Networks for spatial data—as the underlying compressor to further boost performance.
Summary updated