POPE: Post Optimization Posterior Evaluation of Likelihood Free Models

Edward Meeds, Michael Chiang, Mary Lee, Olivier Cinquin, John Lowengrub, Max Welling

Abstract

In many domains, scientists build complex simulators of natural phenomena that encode their hypotheses about the underlying processes. These simulators can be deterministic or stochastic, fast or slow, constrained or unconstrained, and so on. Optimizing the simulators with respect to a set of parameter values is common practice, resulting in a single parameter setting that minimizes an objective subject to constraints. We propose a post optimization posterior analysis that computes and visualizes all the models that can generate equally good or better simulation results, subject to constraints. These optimization posteriors are desirable for a number of reasons among which easy interpretability, automatic parameter sensitivity and correlation analysis and posterior predictive analysis. We develop a new sampling framework based on approximate Bayesian computation (ABC) with one-sided kernels. In collaboration with two groups of scientists we applied POPE to two important biological simulators: a fast and stochastic simulator of stem-cell cycling and a slow and deterministic simulator of tumor growth patterns.

Introduction

In fields ranging from biology to climate science, researchers rely on complex simulators to model natural phenomena. These simulators are often "likelihood-free," meaning there is no simple mathematical formula to describe the probability of an outcome given the parameters. The standard approach is to treat this as an optimization problem: finding the single set of parameters $\theta^*$ that minimizes a cost function. However, this "blind optimization" leaves scientists with several unanswered questions regarding parameter sensitivity, correlations, and potential overfitting to noisy data. Finding a single point in a high-dimensional space doesn't tell you if you're standing on a narrow peak or a broad plateau.

Methodology: Optimization as a Distribution

The core innovation of POPE (Post Optimization Posterior Evaluation) is the adaptation of Approximate Bayesian Computation (ABC) for optimization. Instead of looking for parameters that produce outputs $y$ close to an observation $y^*$, POPE uses one-sided kernels to identify parameters where $y \leq y^*$ (for minimization) or where specific constraints are satisfied.

Key technical components include:

  • One-sided Kernels: Heaviside, Gaussian, or exponential "soft" kernels penalize parameters that fail to meet the optimization target while treating all "better than target" results equally.
  • Adaptive POPE: An online algorithm that updates the target objective $y^*$ and the kernel width $\epsilon$ during the MCMC run, allowing the sampler to hone in on the best regions of parameter space.
  • Response Modeling: To handle expensive or stochastic simulators, the framework incorporates local synthetic likelihoods and can utilize global surrogate models like Gaussian Processes.
Table 1: Posterior statistics of the objective function $y_1$ across different experimental configurations (A-D). The results demonstrate how the target $y^*$, stochasticity $M$, and kernel parameters influence the resulting optimization posterior.

As shown in Table 1, the choice of target $y^*$ and the number of stochastic samples $M$ directly impact the posterior distribution of the objective function. For instance, setting a specific target $y^*$ (Experiment A) centers the posterior mass near that value, whereas unconstrained targets (Experiment B) allow for broader exploration. Increasing $M$ (Experiment D) helps stabilize the posterior in the presence of stochasticity.

Biological Case Studies

The framework was evaluated on two distinct biological simulators:

1. Stem-Cell Niche Geometry

Using a fast, stochastic simulator of C. elegans stem-cell cycling, POPE revealed that while many different niche geometries can produce fast cell cycling, the total number of cells in the niche is the most critical constraint. Interestingly, adding "realistic" constraints (like requiring row sizes to increase monotonically) didn't significantly hurt the best achievable performance but drastically changed the "shape" of the valid parameter space.

2. Colon Cancer Tumor Patterns

In a study of tumor growth patterns using a deterministic simulator, researchers identified three distinct "modes" of spotting patterns that satisfied biological constraints. They found a strong correlation between Wnt decay rates and the number of spots, validating biological hypotheses that blocking Wnt leads to fewer, larger spots. This visualization of trade-offs allowed researchers to identify functionally redundant parameters.

Conclusion and Future Work

POPE moves beyond finding a single "optimal" parameter set to mapping the entire landscape of good solutions. This "optimization posterior" provides automatic sensitivity analysis and reveals hidden correlations between parameters. While challenges remain regarding high dimensionality and computational expense for very slow simulators, POPE represents a significant advancement in making complex simulation-based models more interpretable and robust.

Summary updated

Abstract

In many domains, scientists build complex simulators of natural phenomena that encode their hypotheses about the underlying processes. These simulators can be deterministic or stochastic, fast or slow, constrained or unconstrained, and so on. Optimizing the simulators with respect to a set of parameter values is common practice, resulting in a single parameter setting that minimizes an objective subject to constraints. We propose a post optimization posterior analysis that computes and visualizes all the models that can generate equally good or better simulation results, subject to constraints. These optimization posteriors are desirable for a number of reasons among which easy interpretability, automatic parameter sensitivity and correlation analysis and posterior predictive analysis. We develop a new sampling framework based on approximate Bayesian computation (ABC) with one-sided kernels. In collaboration with two groups of scientists we applied POPE to two important biological simulators: a fast and stochastic simulator of stem-cell cycling and a slow and deterministic simulator of tumor growth patterns.
BibTeX
@article{meeds2015pope,
  author = {Edward Meeds and Michael Chiang and Mary Lee and Olivier Cinquin and John Lowengrub and Max Welling},
  title = {{POPE: Post Optimization Posterior Evaluation of Likelihood Free Models}},
  journal = {BMC Bioinformatics},
  year = {2015},
  doi = {10.1186/s12859-015-0658-1},
}
Figure