5. Optimality criteria for probabilistic numerical methods

Chris J. Oates, Jon Cockayne, Dennis Prangle, T. J. Sullivan, Mark GirolamiOpenalex Unknown 2020 · DOI · arXiv · OpenAlex

Abstract

It is well understood that Bayesian decision theory and average case analysis are essentially identical. However, if one is interested in performing uncertainty quantification for a numerical task, it can be argued that standard approaches from the decision-theoretic framework are neither appropriate nor sufficient. Instead, we consider a particular optimality criterion from Bayesian experimental design and study its implied optimal information in the numerical context. This information is demonstrated to differ, in general, from the information that would be used in an average-case-optimal numerical method. The explicit connection to Bayesian experimental design suggests several distinct regimes in which optimal probabilistic numerical methods can be developed.

Introduction

Probabilistic Numerics (PN) is an emerging field that reinterprets classical numerical tasks—such as integration or solving differential equations—as statistical inference problems. Unlike traditional methods that provide a single point estimate, PN methods return a probability distribution that quantifies uncertainty about the solution. This paper investigates the theoretical foundations of PN, specifically focusing on how to define "optimal" information acquisition for uncertainty quantification (UQ).

Problem Statement: Estimation vs. Uncertainty Quantification

In traditional numerical analysis, the focus is often on "average-case optimal" methods. For instance, the trapezoidal rule is optimal for integration under a Wiener process prior. However, the primary goal of PN is UQ. Standard Bayesian Decision Theory (BDT) focuses on minimizing the expected loss of a single "action" or point estimate. The authors argue that an experiment optimized for a point estimate may not be optimal for producing a high-quality posterior distribution that accurately reflects what remains unknown.

Methodology: BDT and the BPN Criterion

The paper compares two distinct frameworks for choosing "experiments" (e.g., sampling locations):

  1. Bayesian Decision Theory (BDT): The classical approach that seeks to minimize Bayes Risk—the expected loss of the Bayes Act (the point estimate).
  2. Bayesian Experimental Design (BED): A broader framework using utility functions to measure the "usefulness" of information.

The authors propose the BPN Criterion, derived from BED. Unlike BDT, which evaluates the loss of a specific estimate, the BPN utility measures the expected loss between the entire posterior distribution and the true state. For specific metrics, this is equivalent to minimizing the Wasserstein distance between the posterior and a delta mass at the true solution.

Key Insights: The $L^2$ Coincidence

A major contribution of the paper is identifying the conditions under which BDT and BPN converge.

Illustration of the squared-error loss function $\ell(x, a) = (x - a)^2$ which leads to the equivalence of BDT and BPN criteria in Hilbert spaces.

The authors prove (Proposition 3) that if a squared-error loss ($L^2$) is used in an inner product space, the BDT-optimal experiment and the BPN-optimal experiment are identical. This "coincidence" explains why the distinction has often been overlooked in existing PN literature, as many methods implicitly rely on $L^2$ norms. However, when the loss function deviates from squared error (e.g., $L^\infty$ or 0-1 loss), the criteria diverge. An experiment yielding the best point estimate may provide a poor representation of uncertainty.

Results and Applications

The theoretical findings are supported by evaluations in several contexts:

  • Numerical Integration: 1D continuous functions under a Wiener process.
  • Linear Regression: Comparing standard A-, D-, and E-optimality.
  • Partial Differential Equations (PDEs): In a 2D linear elliptic PDE, the authors demonstrate that for $p=2$ (squared error), optimal sampling points form a space-filling design. For $p=\infty$ (maximum error), the BPN-optimal design shifts significantly to prioritize regions that better capture worst-case uncertainty.

Conclusion and Future Directions

While the BPN criterion effectively concentrates posterior mass around the truth, it does not inherently guarantee "calibration"—the statistical validity of credible intervals—which remains dependent on the choice of the prior. Additionally, calculating the BPN criterion involves complex integrals, though Monte Carlo methods offer a path forward. Future work will explore extending these static designs to sequential or adaptive settings, where sampling decisions are made dynamically.

Summary updated

Abstract

It is well understood that Bayesian decision theory and average case analysis are essentially identical. However, if one is interested in performing uncertainty quantification for a numerical task, it can be argued that standard approaches from the decision-theoretic framework are neither appropriate nor sufficient. Instead, we consider a particular optimality criterion from Bayesian experimental design and study its implied optimal information in the numerical context. This information is demonstrated to differ, in general, from the information that would be used in an average-case-optimal numerical method. The explicit connection to Bayesian experimental design suggests several distinct regimes in which optimal probabilistic numerical methods can be developed.
BibTeX
@inproceedings{oates2020optimality,
  author = {Chris J. Oates and Jon Cockayne and Dennis Prangle and T. J. Sullivan and Mark Girolami},
  title = {{5. Optimality criteria for probabilistic numerical methods}},
  booktitle = {Openalex Unknown},
  year = {2020},
  doi = {10.1515/9783110635461-005},
}
Figure