Traditional numerical analysis treats the solution to a mathematical problem—an integral, a linear system, or a differential equation—as a single, deterministic point. However, every time we use a computer to approximate these solutions, we introduce errors through discretization, early termination, or rounding.
In the emerging field of Probabilistic Numerics, researchers argue that these approximation errors should be treated as a source of epistemic uncertainty. By reframing numerical algorithms as statistical inference tasks, we can produce not just a "best guess" solution, but a full posterior distribution that quantifies how much we should trust that result.
Numerical Error as Epistemic Uncertainty
When we solve an Ordinary Differential Equation (ODE) using a standard method like Runge-Kutta, we are essentially performing a sequence of approximations. David Duvenaud, Philipp Hennig, and colleagues point out in Probabilistic ODE Solvers with Runge-Kutta Means that these classical methods provide no measure of confidence. If the step size is too large, the solver might drift significantly from the true trajectory without warning the user.
The core intuition of probabilistic numerics is to place a prior (often a Gaussian Process) over the unknown solution and treat the numerical operations as "data" that update this prior. As Philipp Hennig and Søren Hauberg describe in Probabilistic Solutions to Differential Equations, this turns a deterministic algorithm into a Bayesian inference engine. The resulting "error bars" are not just decorative; they represent the solver's own internal estimate of its approximation error.
Differential Equations as Bayesian Filtering
One of the most powerful realizations in this field is that solving an ODE is mathematically equivalent to a nonlinear Bayesian filtering problem. Filip Tronarp, Simo Särkkä, and their collaborators showed in Probabilistic solutions to ordinary differential equations as nonlinear Bayesian filtering that we can use the machinery of Kalman filters to track the state of an ODE.
In this framework:
- The Prior represents our assumptions about the smoothness of the solution (e.g., a Gauss-Markov process).
- The Prediction Step moves our estimate forward in time according to the dynamics.
- The Update Step treats the evaluation of the derivative function as a "measurement" that corrects our trajectory.
This perspective allows for advanced features like h-adaptivity, where the solver automatically shrinks its step size when uncertainty grows too large, as explored by Nathanael Bosch in Calibrated Adaptive Probabilistic ODE Solvers.
Scaling to High Dimensions and Parallelism
A common critique of Bayesian methods is their computational overhead. However, recent work has brought probabilistic solvers to parity with classical ones. Nicholas Krämer and Philipp Hennig demonstrated in Linear-Time Probabilistic Solution of Boundary Value Problems that state-space inference can solve complex boundary problems in linear time.
Furthermore, these solvers are now breaking the "sequential bottleneck" of time-stepping. Nathanael Bosch and Simo Särkkä introduced ParaIEKS, a parallel-in-time solver that reduces computational complexity from linear to logarithmic ($O(\log N)$) by reframing the problem as a parallel prefix-sum operation. This allows probabilistic solvers to handle systems with millions of dimensions (Probabilistic ODE Solutions in Millions of Dimensions).
Beyond ODEs: Linear Systems and Integration
The probabilistic approach extends to other fundamental numerical tasks:
- Linear Systems ($Ax = b$): When solving massive linear systems, we often stop iterative solvers early to save time. Jon Cockayne and Chris J. Oates developed BayesCG, a probabilistic version of the Conjugate Gradient method. It provides a distribution over the solution vector, quantifying the error introduced by early termination.
- Integration (Quadrature): Toni Karvonen and others have advanced Bayesian Quadrature, which uses Gaussian Processes to integrate functions. Methods like Bayes-Sard Cubature combine the robustness of classical rules with the uncertainty quantification of Bayesian inference.
Why This Matters
The value of probabilistic numerics is most visible in safety-critical applications and complex scientific modeling. In neuroscience, for example, Jonathan Oesterle showed that deterministic solvers can "hide" numerical artifacts that look like genuine biological signals, such as shifted spike times. Probabilistic solvers make these artifacts visible as uncertainty (Probabilistic solvers enable a straight-forward exploration of numerical uncertainty in neuroscience models).
By treating the outputs of our algorithms as probability distributions rather than absolute truths, we can propagate numerical uncertainty through entire computational pipelines, leading to more robust and reliable scientific conclusions.