Abstract
This paper investigates Bayesian inference for parametric statistical models defined by stochastic simulators. While evaluating the likelihood function is often prohibitively expensive, Approximate Bayesian Computation (ABC) provides a framework for approximate inference. Optimisation Monte Carlo (OMC) was recently proposed to accelerate this process via parallel optimization. However, we identify a critical failure mode: OMC generates overconfident approximations by collapsing regions of similar likelihood into single points. We propose Robust OMC (ROMC), an efficient generalization that corrects this by sampling local regions. We demonstrate ROMC's effectiveness on toy examples and complex inverse-graphics tasks.
Introduction: The Challenge of Likelihood-Free Inference
In fields ranging from climate science to computer graphics, researchers rely on complex simulators to model data generation. Performing Bayesian inference—finding the distribution of parameters $\theta$ that could have produced observed data $y$—is difficult because the mathematical likelihood $p(y|\theta)$ is often unavailable or too costly to compute.
Approximate Bayesian Computation (ABC) addresses this by comparing simulated data to observed data. A recent advancement, Optimisation Monte Carlo (OMC), uses optimization to find parameter values that minimize the distance between simulated and observed outputs. While fast, OMC suffers from a significant flaw: it assumes the best-fit point is the only relevant one.
The Failure Mode of OMC
The primary issue with OMC is its tendency toward extreme overconfidence. When the likelihood function is "flat"—meaning a wide range of parameters produces similar results—OMC collapses the entire region into a single point mass. This leads to a posterior approximation that is a narrow spike, failing to represent the true uncertainty of the model.
Robust Optimisation Monte Carlo (ROMC)
To address this "curvature trap," we propose Robust OMC (ROMC). Instead of treating the result of an optimization as a single point, ROMC uses it to define an acceptance region. The process involves three main steps:
- Optimisation: For various random seeds of the simulator, find the parameter values $\theta^*$ that best match the observed data.
- Region Construction: Define a local "proposal region" around each $\theta^*$ where the simulator still produces results within a specified tolerance. This can be done using curvature-based bounding boxes (Boxed ROMC) or Gaussian Process surrogates to define ellipsoids (Ellipsoidal ROMC).
- Local Sampling: Sample multiple points from these regions and weight them appropriately to "fill out" the posterior volume.
By focusing on the volume of the acceptable region rather than just the local gradient, ROMC remains robust even when the likelihood is flat or the summary statistics are uninformative.
Experimental Evaluation
We evaluated ROMC across three distinct scenarios:
- 1D Toy Simulator: Specifically designed to trigger OMC's collapse by featuring a flat likelihood.
- Normal Distribution Inference: A benchmark where summary statistics are intentionally incomplete, testing the method's ability to handle uninformative data directions.
- Inverse Graphics (Teapot): A high-dimensional task (20 parameters) using the
OpenDRrenderer to infer color, lighting, and pose from a single image. This task is inherently bi-modal (e.g., a red teapot under white light vs. a gray teapot under red light).
Key Results and Performance
ROMC significantly outperforms OMC in both accuracy and uncertainty quantification. In the inverse graphics task, while OMC collapsed to a single incorrect solution, ROMC successfully identified both possible modes of the posterior.
Quantitatively, ROMC achieves much lower Jensen-Shannon (JS) divergence from the ground truth compared to OMC and other heuristic fixes. Furthermore, ROMC maintains a high Effective Sample Size (ESS), whereas OMC's ESS drops precipitously in the presence of flat likelihoods.
Conclusion and Future Work
Robust OMC provides a necessary correction to the overconfidence of standard OMC while retaining its computational benefits. By intelligently sampling local regions, it captures the true breadth of the posterior distribution. Future work will focus on scaling ROMC to even higher-dimensional target parameters and exploring more flexible, non-linear proposal regions to better approximate complex likelihood landscapes.
Summary updated