A Parzen-Based Distance Between Probability Measures as an Alternative of Summary Statistics in Approximate Bayesian Computation

Carlos D. Zuluaga, Edgar A. Valencia, Mauricio A. Álvarez, Álvaro Á. Orozco

TL;DR

This paper introduces a new, more robust method for Approximate Bayesian Computation (ABC) called PABC. It improves upon existing techniques by using a smoother way to compare data distributions, making it more reliable, especially when dealing with limited data. This advancement is crucial for Bayesian inference in complex models where direct likelihood calculation is impossible.

What Problem Does This Paper Address?

Approximate Bayesian Computation (ABC) is a powerful set of techniques for performing Bayesian inference when the likelihood function of a model is either computationally expensive or impossible to evaluate. The core idea of ABC is to simulate data from a model using different parameter values and then compare these simulated datasets to the observed data. This comparison is typically done by calculating a distance between summary statistics extracted from both the simulated and observed data.

A major challenge in ABC is selecting appropriate summary statistics. Poorly chosen statistics can lead to biased posterior distributions or inefficient sampling. While recent advances have proposed nonparametric ABC methods that avoid manual summary statistic selection by using kernel embeddings to compare distributions, these methods often assume discrete data distributions. This assumption can be problematic and lead to a lack of robustness, particularly when dealing with small datasets. This paper aims to address this limitation by proposing a more robust nonparametric ABC method that can handle continuous data distributions.

Methods

The paper introduces a new metric for comparing probability distributions within a Reproducing Kernel Hilbert Space (RKHS). Unlike previous nonparametric ABC methods that assumed discrete distributions (e.g., using empirical distributions), this new approach, termed PABC (Parzen-based Approximate Bayesian Computation), utilizes a Parzen-window density estimator to obtain smooth estimates of the probability density functions.

This smoother estimation allows PABC to compare continuous probability distributions more effectively. The core of the method involves calculating a dissimilarity distance between these estimated densities. This distance is a combination of a Maximum Mean Discrepancy (MMD) term and a term related to the kernel embedding itself, weighted by a parameter $\lambda$. The authors also propose using Gaussian kernels for this comparison.

The paper also extends these methods to the Sequential Monte Carlo (SMC) framework, leading to PABC SMC. This extension aims to improve efficiency and refine the posterior approximation over a series of steps, especially for complex models.

Datasets

The paper evaluates its proposed methods on two datasets:

  • Synthetic Dataset: Generated from an uniform mixture model. This dataset is used to test the accuracy and robustness of the methods, particularly in relation to the number of observations.
  • Real Dataset: A time series of adult blowfly population data, previously used in other ABC research. This dataset allows for the evaluation of the methods on a real-world ecological dynamic system.

Key Insights and Ideas

  • Smoothness is Key for Robustness: By moving from discrete empirical distributions to smooth density estimates (Parzen windows), the proposed method (PABC) becomes significantly more robust, especially when dealing with limited data. This is a critical insight for practical ABC applications.
  • Bridging the Gap: PABC effectively bridges the gap between traditional summary-statistic-based ABC and purely nonparametric kernel-based methods by offering a flexible and robust way to compare distributions.
  • SMC Extension Enhances Efficiency: The integration of PABC with Sequential Monte Carlo (SMC) further improves the efficiency and accuracy of the posterior inference, making it suitable for more complex and challenging problems.
  • Parameter Sensitivity in Complex Models: The results on the blowfly dataset highlight that for nonlinear dynamic systems exhibiting chaotic behavior, certain parameters can be highly sensitive, leading to less similar posteriors across different ABC methods.

Key Results

  • Synthetic Data:
    • PABC demonstrated the steadiest and lowest Root Mean Squared Error (RMSE) across varying numbers of observations, indicating superior robustness compared to ABC and K2ABC.
  • For instance, PABC achieved a mean RMSE of 0.0696 ± 0.0006, outperforming ABC (0.0879 ± 0.0050) and K2ABC (0.0733 ± 0.0031).
  • PABC SMC also showed low variability and good performance.
  • Real Data (Blowfly Population):
    • PABC and PABC SMC showed the highest median cross-correlation coefficient ($\rho$) between simulated and observed time series, indicating better model fit and predictive performance.
  • PABC achieved a median $\rho$ of 0.6501, outperforming K2ABC (0.6138) and ABC (0.6138).
  • The posterior distributions for some parameters in the nonlinear ecological system were similar across methods, while others showed divergence, particularly for parameters related to the system's dynamic range.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the proposed estimator for the dissimilarity distance is biased, although it demonstrates a lower root mean squared error compared to the MMD estimator in certain contexts.
  • While PABC is more robust than previous discrete-distribution-based nonparametric ABC methods, the paper doesn't explicitly detail its performance limitations with extremely sparse data.
  • Future Work:
    • The authors suggest exploring new dissimilarity distances using RKHS for other applications, such as electrical network analysis.
  • The potential for applying this smoother density estimation approach to other complex models and time-series analysis problems is implied.

Summary updated

Abstract

Approximate Bayesian Computation (ABC) are likelihood-free Monte Carlo methods. ABC methods use a comparison between simulated data, using different parameters drawn from a prior distribution, and observed data. This comparison process is based on computing a distance between the summary statistics from the simulated data and the observed data. For complex models, it is usually difficult to define a methodology for choosing or constructing the summary statistics. Recently, a nonparametric ABC has been proposed, that uses a dissimilarity measure between discrete distributions based on empirical kernel embeddings as an alternative for summary statistics. The nonparametric ABC outperforms other methods including ABC, kernel ABC or synthetic likelihood ABC. However, it assumes that the probability distributions are discrete, and it is not robust when dealing with few observations. In this paper, we propose to apply kernel embeddings using a sufficiently smooth density estimator or Parzen estimator for comparing the empirical data distributions, and computing the ABC posterior. Synthetic data and real data were used to test the Bayesian inference of our method. We compare our method with respect to state-of-the-art methods, and demonstrate that our method is a robust estimator of the posterior distribution in terms of the number of observations.
BibTeX
@inproceedings{zuluaga2015parzen,
  author = {Carlos D. Zuluaga and Edgar A. Valencia and Mauricio A. Álvarez and Álvaro Á. Orozco},
  title = {{A Parzen-Based Distance Between Probability Measures as an Alternative of Summary Statistics in Approximate Bayesian Computation}},
  booktitle = {ICIAP},
  year = {2015},
  doi = {10.1007/978-3-319-23231-7\_5},
}
Figure