Neural Approximate Sufficient Statistics for Implicit Models

Yanzhi Chen, Dinghuai Zhang, Michael U. Gutmann, Aaron C. Courville, Zhanxing Zhu

Abstract

We consider the fundamental problem of how to automatically construct summary statistics for implicit generative models where the evaluation of the likelihood function is intractable, but sampling data from the model is possible. The idea is to frame the task of constructing sufficient statistics as learning mutual information maximizing representations of the data with the help of deep neural networks. The infomax learning procedure does not need to estimate any density or density ratio. We apply our approach to both traditional approximate Bayesian computation and recent neural likelihood methods, boosting their performance on a range of tasks.

Introduction

Implicit generative models, such as simulators used in epidemiology or cosmology, are powerful tools for scientific modeling. However, performing Bayesian inference on these models is challenging because their likelihood functions are often intractable. Likelihood-Free Inference (LFI) methods, like Approximate Bayesian Computation (ABC) and Neural Likelihood (NL), address this by comparing simulated data to observed data.

A significant bottleneck in LFI is the "curse of dimensionality," which necessitates compressing high-dimensional raw data into low-dimensional summary statistics. Traditionally, these statistics are hand-crafted by domain experts. If these statistics are not "sufficient"—meaning they fail to capture all relevant information about the model parameters—the resulting inference can be biased or overconfident. This paper introduces a framework for automatically learning globally sufficient statistics using neural networks.

Methodology

The core contribution of this work is the use of the "infomax" principle to learn summary statistics. A statistic $s(x)$ is sufficient for parameters $\theta$ if and only if it maximizes the Mutual Information (MI) between $\theta$ and $s(x)$. The authors propose training a "statistic network" to map raw data to a low-dimensional vector by maximizing an MI estimator.

Comparison of traditional expert-driven LFI (left) and the proposed neural infomax approach (right).

Two primary MI estimators are explored:

  1. Jensen-Shannon Divergence (JSD): A robust estimator that requires an auxiliary "critic" network.
  2. Distance Correlation (DC): A "ratio-free" estimator that is significantly faster and more efficient for large-scale tasks.

The authors also introduce a multi-round strategy where the statistic network and the posterior solver are refined iteratively, ensuring that the learned statistics are optimized for the current posterior estimate. This leads to enhanced versions of existing algorithms: SMC-ABC+ and SNL+.

Experiments and Results

The method was evaluated on three distinct data types: spatial data (Ising model), i.i.d. data (Gaussian Copula), and time-series data (Ornstein-Uhlenbeck process). These models were chosen because their true posteriors are either known or can be computed through intensive numerical methods, providing a ground truth for evaluation.

Evaluation on the Ising model: (a) sample grid data, (b) performance comparison (JSD) across rounds, and (c) correlation between learned neural statistics and true sufficient statistics.

Key findings include:

  • Accuracy: The "plus" versions of the algorithms (SMC-ABC+ and SNL+) consistently outperformed standard baselines and original versions using raw data.
  • Sufficiency: As shown in the Ising model results, the learned neural statistics correlate strongly with the known sufficient statistics, confirming the theoretical link between MI maximization and sufficiency.
  • Efficiency: The DC estimator proved to be approximately 15 times faster per mini-batch than other MI estimators, making it highly practical for complex models.
  • Versatility: Simple fully-connected architectures were effective across different data structures, though specialized architectures (e.g., GNNs or Transformers) could likely yield even better results.

Conclusion

By framing the construction of summary statistics as a mutual information maximization problem, this work provides a principled and automated way to perform high-quality Likelihood-Free Inference. The approach eliminates the need for manual feature engineering and scales well to complex, high-dimensional data, bridging the gap between simulator-based modeling and robust Bayesian analysis.

Summary updated

Abstract

We consider the fundamental problem of how to automatically construct summary statistics for implicit generative models where the evaluation of the likelihood function is intractable, but sampling data from the model is possible. The idea is to frame the task of constructing sufficient statistics as learning mutual information maximizing representations of the data with the help of deep neural networks. The infomax learning procedure does not need to estimate any density or density ratio. We apply our approach to both traditional approximate Bayesian computation and recent neural likelihood methods, boosting their performance on a range of tasks.
BibTeX
@inproceedings{chen2021neural,
  author = {Yanzhi Chen and Dinghuai Zhang and Michael U. Gutmann and Aaron C. Courville and Zhanxing Zhu},
  title = {{Neural Approximate Sufficient Statistics for Implicit Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=rbWihvoje5},
}
Figure