Abstract
Proceedings of the National Academy of Sciences (PNAS), a peer reviewed journal of the National Academy of Sciences (NAS) - an authoritative source of high-impact, original research that broadly spans the biological, physical, and social sciences. This paper critically evaluates the proposal to treat approximation errors in Approximate Bayesian Computation (ABC) as inferable parameters, highlighting fundamental issues in identifiability and Bayesian logic.
Introduction
In the realm of Approximate Bayesian Computation (ABC), researchers often grapple with the "likelihood-free" problem, where the likelihood function is too complex to calculate directly. The standard approach involves using a small "tolerance" or error term ($\epsilon$) to accept simulations that are "close enough" to the observed data. Traditionally, $\epsilon$ has been viewed as a computational nuisance parameter that must be carefully calibrated.
However, a provocative 2009 paper by Ratmann et al. proposed a "change of perception": treating $\epsilon$ as a formal parameter of the model to be inferred. By doing so, they suggested that one could perform "model criticism" by observing the posterior distribution of this error term. This paper by Robert, Mengersen, and Chen provides a rigorous statistical critique of this shift, arguing that it introduces significant theoretical and practical complications.
The Identifiability Trap and Prior Influence
A primary concern raised by the authors is the "Identifiability Trap." In many common statistical models, such as location families, the error term $\epsilon$ and the model parameter $\theta$ are not jointly identifiable from the data. This means that the data cannot distinguish between a poorly chosen parameter value and a fundamentally flawed model.
Mathematically, the authors demonstrate that the posterior distribution of the error term often simply returns the prior distribution, providing no new information from the data. Consequently, the results of such "model criticism" are dangerously sensitive to the choice of the supplementary prior ($\pi_\epsilon$).
Theoretical Flaws in the Likelihood Construction
The authors identify a technical but crucial flaw in the Ratmann approach: the "likelihood" function defined for the error term is not a true density in the space of observations. This lack of a proper density means the framework is not invariant under reparameterization, a fundamental requirement for rigorous Bayesian inference.
Furthermore, the approach lacks a built-in "complexity penalty." Standard Bayesian model choice, typically performed using Bayes factors, naturally incorporates Occam's Razor by penalizing models with excessive parameters. The authors argue that comparing models based on error distributions lacks this inherent penalization, which could potentially favor overly complex models that "fit" the error better without being better models of the underlying process.
Sensitivity and Model Mismatch
To test the logic of the "error-as-parameter" framework, the authors applied it to several "toy models," including Poisson and binomial distributions. In the Poisson case, they discovered a "puzzling non-monotonicity" in the derived p-values. As the observed data became more extreme (and thus less likely under the model), the p-value did not always decrease consistently, making it an unreliable metric for model rejection.
Crucially, the authors demonstrate a "sensitivity gap" between standard Bayesian evidence and the proposed p-value. As shown in the figure below, the Bayes factor (marginal likelihood) is much more sensitive to model inadequacy than the proposed p-value. The evidence drops toward zero much faster than the p-value as the data deviates from the model's predictions, suggesting that standard Bayesian methods are more robust for detecting model mismatch.
In the binomial model example, the authors went further to prove mathematically that the data provides zero information about the error term $\epsilon$. This renders the entire inference process on the error moot, as the posterior is identical to the prior regardless of the observed data.
Conclusion
While the authors acknowledge that the Ratmann approach might serve as an "informal assessment" tool within the ABC community, they strongly warn against treating it as a replacement for formal Bayesian model choice. The method's reliance on non-identifiable parameters and its sensitivity to prior choices undermine its reliability. Moreover, while the method focuses on model rejection, it fails to provide a clear Bayesian path for model improvement or selection among alternatives. For rigorous scientific inference, the authors maintain that standard Bayesian evidence remains the superior tool for model evaluation.
Summary updated