Sample Path Regularity of Gaussian Processes

19 concepts6 papers2 proposals

When using Gaussian Processes (GPs) or infinitely wide neural networks, we often select a covariance kernel based on a vague intuition about "smoothness." We know that a Squared Exponential kernel produces infinitely differentiable functions, while a Matérn kernel yields rougher, more jagged surfaces.

However, in rigorous applications—such as solving differential equations, modeling spatial probability distributions, or warping medical images—intuition is not enough. If the functions drawn from a GP prior (its sample paths) are smoother or rougher than the true underlying physical process, the model can fail to converge, produce invalid mathematical operations, or output dangerously over-confident uncertainty estimates.

Recent mathematical developments have bridged this gap, establishing exact, dictionary-like mappings between the algebraic structure of a kernel and the topological smoothness of its sample paths.

The Gap Between Learning and Drawing

To understand why sample path regularity matters, we must first distinguish between two spaces:

  1. The Reproducing Kernel Hilbert Space (RKHS): This is the space of functions that the GP can theoretically learn or approximate as data arrives.
  2. The Sample Path Space: This is the space where individual random functions drawn from the GP prior actually live.

Intuitively, one might assume these two spaces share the same level of smoothness. In reality, they consistently diverge.

As Holzmüller and Schölpple demonstrate in the context of wide neural networks at initialization, functions sampled from the network's Gaussian process limit are consistently smoother than the functions the network is theoretically capable of learning. Specifically, these initialization sample paths possess a higher degree of regularity—half an order more in Sobolev space terms—than the functions within the network's RKHS.

This distinction is crucial. If a variational framework or an approximation method (such as Fourier-based features) is built assuming the sample paths share the exact regularity of the RKHS, the underlying mathematical assumptions can be violated, rendering the approximation ill-defined (Wynne et al.).

Mapping Kernels to Exact Smoothness Spaces

How do we precisely determine the smoothness of a sample path just by looking at its kernel? Historically, researchers relied on loose sufficient conditions. A foundational framework by Da Costa et al. resolved this by establishing necessary and sufficient conditions that map covariance kernels directly to specific local Hölder and Sobolev spaces.

Hölder regularity dictates how radically a function can fluctuate over short distances, while Sobolev regularity extends this to weak differentiability (the existence of well-behaved derivatives in an average sense).

For the widely used Matérn class of Gaussian Processes, the authors prove a tight characterization: the sample paths exhibit a precise level of continuous differentiability directly tied to the smoothness parameter $\nu$. Specifically, they are continuously differentiable exactly $\lfloor \nu \rfloor$ times, and absolutely no more.

Kernel Choice (e.g., Matérn ν) 
       │
       ▼ (Da Costa et al. Central Theorem)
Exact Sample Path Regularity (e.g., ⌊ν⌋ times continuously differentiable)
       │
       ├──────────────────────────────┼──────────────────────────────┐
       ▼                              ▼                              ▼
Probabilistic Numerics        Spatial Modeling              Image Registration
(Match PDE constraints)       (Preserve TV/Hellinger)       (Enforce physical warping)

This mathematical precision allows practitioners to move away from trial-and-error kernel selection and instead perform differentiability-constrained prior elicitation.

Why Regularity Dictates Practical Success

When a probabilistic model's sample paths do not match the physical reality of the problem, the consequences manifest as broken calibration or structural failures.

1. Calibrating Uncertainty in Numerical Solvers

In probabilistic numerics, physical laws are treated as inference problems. Wang et al. apply local linearization and GP priors to solve nonlinear partial differential equations (PDEs). They show that if you choose a kernel that assumes infinite smoothness—such as the Rational Quadratic or Squared Exponential kernel—the resulting numerical solver becomes severely over-confident and yields poorly calibrated uncertainty estimates.

Because a PDE dictates strict differentiability requirements, the prior must match that exact space. By utilizing Matérn kernels with finite, specified differentiability tailored to the PDE's order, the solver correctly represents the true solution space and provides robust, trustworthy error quantification.

2. Preserving Distances in Spatial Fields

When modeling how entire probability distributions evolve across a geographic map, the transitions must be continuous. Gautier and Ginsbourger developed Spatial Logistic Gaussian Processes (SLGPs) to predict complex density fields.

They proved that the spatial regularity of the predicted density field—measured by statistical distances like Hellinger or Total Variation—is directly inherited from the covariance kernel of the underlying GP. By selecting specific kernel properties governing Hölder continuity, practitioners can precisely control how rapidly or smoothly a probability distribution shifts from one spatial coordinate to the next.

3. Maintaining Physical Realism in Image Warping

In medical image registration, where one anatomy is "warped" to match another, the deformation field cannot contain tearing or unrealistic discontinuities. Bayer et al. leverage a squared exponential kernel to define the spatial covariance of landmarks. Because the kernel guarantees infinite mathematical smoothness, the resulting transformation field maintains strict physical regularity, while simultaneously providing clinicians with a pixel-by-pixel uncertainty map of the deformation.

Conclusion

Sample path regularity is not an abstract mathematical nicety; it is the mechanism that governs how a Gaussian process behaves in the wild. Whether ensuring that a neural network's initialization bias is well-understood, guaranteeing that a spatial density field changes continuously, or preventing a PDE solver from hallucinating precision, matching the kernel's mathematical regularity to the problem's physical constraints is essential for building reliable, calibrated probabilistic models.

Key Foundational Work

Bayesian Numerical Methods for Nonlinear Partial Differential Equations (2021) — contributed 3 concept elements to this direction.
Continuous logistic Gaussian random measure fields for spatial distributional modelling (2021) — contributed 2 concept elements to this direction.
An Investigation of Feature-based Nonrigid Image Registration using Gaussian Process (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sample_path_smoothness_discrepancy_2b2fe06d,
  title = {sample path smoothness discrepancy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sample-path-smoothness-discrepancy/rdr_2b2fe06d146a42e8be3d1e5342dfcb7b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_sample_path_smoothness_discrepancy_2b2fe06d_v1,
  title = {sample path smoothness discrepancy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sample-path-smoothness-discrepancy/rdr_2b2fe06d146a42e8be3d1e5342dfcb7b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake