Quantifying Discretization Error with Probabilistic Solvers

66 concepts49 papers1 proposal

When we use a computer to solve a differential equation, invert a massive matrix, or compute an integral, we are accustomed to receiving a single, deterministic answer. Classical numerical analysis treats these tasks as purely algebraic or algorithmic exercises. However, because digital computers can only evaluate functions at a finite number of points or run for a finite number of iterations, the true mathematical solution remains fundamentally unobserved.

As noted by Girolami et al., this finite computational budget introduces a distinct form of uncertainty—epistemic numerical uncertainty. When a deterministic solver hides this error, downstream applications treat an approximation as absolute truth. This can lead to overconfident predictions, miscalibrated risks, or optimization algorithms getting trapped by numerical artifacts (Teymur et al.; Bartels et al.).

The field of probabilistic numerics solves this problem by reframing numerical algorithms as statistical inference engines. Instead of asking "What is the point estimate of this mathematical quantity?" it asks "What is the posterior probability distribution over the true solution, given the computational work we have performed?"


The Core Mechanism: Computation as Inference

To understand why this perspective works, we can map the components of a standard numerical task directly onto the machinery of Bayesian inference, as formalized by Fitzsimons et al.:

  1. The Prior: We encode our structural assumptions about the latent mathematical object. For instance, if we are solving a differential equation, we might place a Gaussian process (GP) prior over the trajectory to express how smooth or differentiable we expect the solution to be.
  2. The Data: Every time the algorithm performs a computational step—such as evaluating a derivative or multiplying a vector by a matrix—it generates a new piece of deterministic "data."
  3. The Likelihood and Posterior: We treat these computational outputs as noise-free or noisy observations of the latent solution. By applying Bayes' rule, we update our prior into a posterior distribution.

The mean of this posterior distribution typically corresponds to a classical numerical estimate, while its variance provides a calibrated "error bar" representing the remaining numerical uncertainty.


Probabilistic ODE Solvers via State-Space Filtering

Ordinary differential equations (ODEs) of the form $\dot{x}(t) = f(x(t), t)$ are traditionally solved using step-by-step integration methods like Runge-Kutta. Duvenaud et al. demonstrated that these classical algorithms can be interpreted as Gaussian process regression, allowing researchers to construct solvers that output the exact same point estimates as standard methods but with a principled measure of confidence.

To make this inference computationally efficient, Tronarp et al. reframed ODE integration as a nonlinear Bayesian filtering and smoothing problem. By modeling the latent trajectory as a Gauss-Markov process, the solver can step through time using Kalman-like updates.

[Prior Belief: Smooth Trajectory] 
               │
               ▼
┌──────────────────────────────┐
│  Evaluate Derivative f(x,t)  │ ◄─── This acts as a "measurement"
└──────────────┬───────────────┘
               │
               ▼
[Kalman Filter Update Step] ───► Updates Mean (Solution) & Variance (Error)

At each discrete time step, evaluating the vector field $f(x(t), t)$ provides a localized measurement of the system's dynamics. The filtering equations then propagate both the estimated state and its covariance forward in time. When the vector field is highly non-linear or "stiff," standard explicit probabilistic solvers can struggle; however, Bosch et al. resolved this by introducing probabilistic exponential integrators that embed the linear components of the system directly into the prior, ensuring numerical stability alongside uncertainty quantification.


Probabilistic Linear Solvers

The same inference paradigm applies to linear algebra. Consider the problem of solving $Ax = b$ for a massive, sparse matrix $A$. Iterative methods like the Conjugate Gradient (CG) algorithm generate a sequence of vectors that move closer to the true solution with each iteration.

In Probabilistic Interpretation of Linear Solvers, Hennig proved that classical CG is mathematically equivalent to performing Bayesian inference under a specific prior over the matrix's properties. Every matrix-vector multiplication acts as a linear observation of $A$ or its inverse.

When computational constraints force us to stop an iterative solver early, a probabilistic linear solver does not merely return the final iteration's vector; it provides a posterior distribution over the remaining error vector (Reid). This allows practitioners to treat preconditioning not as an arbitrary algorithmic trick, but as a formal prior belief about the matrix structure (Bartels et al.).


Overcoming the Cost of Inference

The primary critique of probabilistic numerics has historically been its computational overhead: if a Gaussian process requires $O(N^3)$ operations, why spend more time computing error bars than solving the original problem?

Recent breakthroughs have eliminated this bottleneck by exploiting structural properties in the underlying mathematics:

  • High-Dimensional Systems: For ODEs spanning millions of dimensions, Krämer et al. achieved linear $O(d)$ scaling by enforcing independence assumptions and Kronecker-structured covariance matrices in the prior models.
  • Partial Differential Equations (PDEs): For spatial domains governed by PDEs, Weiland et al. replaced dense Gaussian processes with Gaussian Markov Random Fields (GMRFs). Because GMRFs rely on sparse precision matrices, they match the speed of classical finite-element methods while retaining full Bayesian uncertainty tracking.

Why It Matters: Propagating Error Downstream

In complex machine learning and simulation pipelines, numerical errors are frequently ignored, leaving them out of the total uncertainty budget (Probabilistic Linear Solvers for Machine Learning). This omission can have severe consequences when fitting parameters to physical systems.

For instance, when optimizing the parameters of a dynamical system based on noisy observations, standard optimization algorithms can easily be misled by the rugged, artificial landscapes created by discretization errors. By incorporating the solver's epistemic uncertainty directly into the likelihood function, Kersting et al. demonstrated that optimization routines become robust to these artifacts.

Taking this a step further, Beck et al. introduced diffusion tempering. This technique intentionally inflates the numerical uncertainty at the beginning of an optimization task to smooth out local minima in the loss landscape. As the optimization progresses, the solver's step size is refined, gradually reducing the numerical variance and guiding gradient descent reliably toward the true global optimum.

Key Foundational Work

Probabilistic Solutions to Differential Equations and their Application to Riemannian Statistics (2014) — contributed 2 concept elements to this direction.
Probabilistic ODE Solvers with Runge-Kutta Means (2014) — contributed 1 concept elements to this direction.
Probabilistic Numerical Analysis in Riemannian Statistics (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_numerical_uncertainty_quantification_908da56b,
  title = {numerical uncertainty quantification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/numerical-uncertainty-quantification/rdr_908da56b36a442cc8d84bc92244b9807},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_numerical_uncertainty_quantification_908da56b_v1,
  title = {numerical uncertainty quantification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/numerical-uncertainty-quantification/rdr_908da56b36a442cc8d84bc92244b9807/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake