Standard Gaussian Process Regression (GPR) is a cornerstone of Bayesian machine learning, celebrated for its elegant formulation and well-calibrated uncertainty estimates. However, standard GPR suffers from a fundamental limitation: stationarity.
A stationary kernel assumes that the correlation between any two points depends solely on their relative distance vector $\tau = x - x'$. This implies that the function's smoothness, or "wiggliness," is governed by a single, global lengthscale parameter $\ell$ across the entire input space.
If you are modeling a landscape containing both flat, predictable plains and jagged, volatile mountain peaks, a stationary kernel forces an unacceptable compromise. Choosing a small lengthscale allows the model to capture the sharp transitions of the peaks but causes it to overfit and over-oscillate in the plains. Choosing a large lengthscale smooths out the noise in the plains but entirely erases the peaks. As noted by Ingmar Posner et al. in Efficient Non-Parametric Surface Representations Using Active Sampling for Push Broom Laser Data and Mike Smith in Adaptive Data Compression for Robot Perception, this uniform texture assumption fails when modeling real-world environments that feature heterogeneous spatial structures.
To solve this, a rich body of research has shifted toward non-stationary kernel modeling, where the hyperparameters themselves—such as lengthscale and variance—vary dynamically as a function of the input location.
Constructing Valid Non-Stationary Kernels
You cannot simply swap a constant lengthscale $\ell$ for a function $\ell(x)$ in a standard Squared Exponential kernel. Doing so typically violates a core mathematical requirement of Gaussian Processes: the resulting covariance matrix must be positive semi-definite (PSD).
The foundational breakthrough to overcome this constraint was developed by Christopher J. Paciorek et al. in Nonstationary Covariance Functions for Gaussian Process Regression. The authors derived a mathematically rigorous class of non-stationary covariance functions—including a non-stationary Matérn kernel—by allowing local kernel matrices to vary across input locations.
A prominent manifestation of this approach is the Gibbs kernel, highlighted by Vidhi Lalchand et al. in Kernel Learning for Explainable Climate Science. In a Gibbs kernel, the covariance between two points $x$ and $x'$ scales with an average of their local lengthscales $\ell(x)$ and $\ell(x')$, modified by a prefactor that preserves the PSD property:
$$K(x, x') \propto \left( \frac{2\ell(x)\ell(x')}{\ell^2(x) + \ell^2(x')} \right)^{1/2} \exp\left( -\frac{(x - x')^2}{\ell^2(x) + \ell^2(x')} \right)$$
Intuitively, when $x$ and $x'$ are both in a highly volatile region, their local lengthscales are small, causing the correlation to drop off rapidly with distance. When they reside in a smooth region, the lengthscales expand, allowing the correlation to persist over longer distances.
Another intuitive perspective is input space warping. Instead of altering the kernel equation, you warp the underlying geometry of the input space itself. If you stretch the input space in volatile regions and compress it in smooth regions, a standard stationary kernel applied to this warped space will behave non-stationarily relative to the original coordinates.
For example, David Frich Hansen et al. in Probabilistic Signal Estimation for Vibrational Spectroscopy with a Flexible Non-Stationary Gaussian Process Baseline Model apply a soft window function to stretch the input space specifically beneath spectral peaks, forcing the baseline model to become locally smooth and linear so it does not absorb the signal. Similarly, Eric Dexheimer et al. show in Learning a Depth Covariance Function that a learned transformation can warp pixel distances to ensure pixels within the same object remain highly correlated, while pixels across geometric boundaries become independent.
How to Parameterize and Learn Non-Stationarity
Introducing a spatially varying lengthscale function $\ell(x)$ creates a secondary problem: how do we learn an infinite-dimensional function from finite data? Researchers have developed three main paradigms to parameterize this non-stationarity.
1. Hierarchical Gaussian Processes
The most natural Bayesian approach is to treat the lengthscale as a latent process governed by a secondary, auxiliary Gaussian Process. Sahil Garg in Learning Non-Stationary Space-Time Models for Environmental Monitoring and Zeel B. Patel et al. in Accurate and Scalable Gaussian Processes for Fine-Grained Air Quality Inference utilize this hierarchical setup to let the model learn how correlation scales shift dynamically across space and time.
To make this computationally practical, Christian Plagemann et al. demonstrate in Nonstationary Gaussian Process Regression Using Point Estimates of Local Smoothness that formulating these non-stationary covariance functions to be fully differentiable allows for the joint optimization of data parameters and local lengthscales using fast, gradient-based techniques.
2. Neural Network Parameterization
Deep neural networks can serve as powerful function approximators to output local kernel parameters directly from coordinates. Alexander Bukharin et al. leverage this in Early Detection of COVID-19 Hotspots Using Spatio-Temporal Data, using a neural network to dynamically output local covariance matrices based on geographic coordinates. This allows the model to capture complex, non-stationary transmission dynamics across urban hubs and rural counties.
In a similar vein, Jean-François Ton et al. combine deep learning with Fourier feature approximations in Spatial Mapping with Gaussian Processes and Nonstationary Fourier Features to learn localized textures without incurring the heavy computational overhead typically associated with non-stationary models. This connection between deep architectures and non-stationarity is deeply rooted; as Christopher K. I. Williams proved in Computing with Infinite Networks, infinite neural networks with specific weight priors inherently generate non-stationary kernels where correlations depend on absolute coordinates rather than relative distance.
3. Process Convolutions and Spectral Methods
A third approach constructs non-stationarity by viewing a Gaussian Process as a white noise process passed through a continuous smoothing filter (a convolution). By allowing the parameters of this filter to vary across the input domain, non-stationarity emerges naturally.
Magnus Ross et al. utilize this in Nonparametric Gaussian Process Covariances via Multidimensional Convolutions to automatically identify inhomogeneous spatial structures without manual kernel engineering. Kangrui Wang et al. extend this concept in Non-separable Non-stationary random fields via Stochastic Process Mixing, which uses infinite mixtures of convolved processes to model changing spatial and temporal dependencies.
In the frequency domain, Hui Xue et al. in Deep Spectral Kernel Learning and Jian Li et al. in Convolutional Spectral Kernel Learning with Generalization Guarantees (Abstract Reprint) leverage non-stationary spectral kernels derived via Yaglom’s theorem, allowing similarity measures to vary across different regions of an input space to capture complex, long-range interdependencies.
Alternative Paradigms: Mixtures and Partitioning
Instead of defining a single, continuously varying non-stationary kernel, some frameworks adopt a "divide-and-conquer" strategy to localize stationarity.
- Mixtures of Experts: Volker Tresp pioneered this in Mixtures of Gaussian Processes, combining multiple localized GP experts so that high-bandwidth experts automatically handle volatile regions while low-bandwidth experts model smooth regions. Similarly, Noémie Jaquier et al. show in Learning from demonstration with model-based Gaussian process that a non-stationary kernel can be constructed by summing multiple kernels weighted by the responsibilities of a Gaussian Mixture Model (GMM).
- Tree-Based Partitioning: Tamara Broderick et al. demonstrate in Classification and categorical inputs with treed Gaussian process models that a recursive tree structure can partition the input space into distinct regions. By fitting independent stationary GPs within each leaf node, the model successfully captures abrupt, non-stationary changes across boundaries.
- Localized Basis Functions: Rather than relying on global parameters, Ibrahim Almosallam et al. show in A sparse Gaussian process framework for photometric redshift estimation that assigning unique hyperparameters to individual basis functions allows a sparse GP to adapt efficiently to local data patterns. For massive datasets, Matthias Katzfuß demonstrates in Bayesian nonstationary spatial modeling for very large datasets that modeling spatial parameters like smoothness and variance as linear combinations of basis functions provides robust uncertainty quantification in heterogeneous environments.
Why It Works: Downstream Impacts on Uncertainty and Search
The primary benefit of non-stationary kernel modeling is not just better curve fitting—it is the dramatic improvement in uncertainty calibration.
In standard Bayesian Optimization (BO), a stationary surrogate model assumes uniform behavior across the search space. This causes the acquisition function to over-explore flat, unpromising reward landscapes or under-explore highly localized, peaky regions.
Ian Char et al. address this in Offline Contextual Bayesian Optimization by allowing the lengthscale to vary with task context, preventing wasteful over-exploration. Afonso Eduardo et al. show in Bayesian Optimization with Informative Covariance that varying expected variance and lengthscale allows researchers to bake spatial prior beliefs directly into the GP, focusing search resources on promising regions.
Furthermore, when optimizing over bounded domains, standard kernels suffer from edge effects. Han Zhou solves this in Bayesian Optimization over Bounded Domains with the Beta Product Kernel by designing a non-stationary Beta kernel that naturally adapts its correlation structure based on absolute position, accurately capturing changing uncertainty near the edges and corners of a hypercube.
Stationary GP: [ Smooth Region ] ---> ( Global Lengthscale ℓ ) ---> [ Volatile Region ]
|
(Over-smooths peaks)
Non-Stationary: [ Smooth Region ] ---> ( Large Local ℓ(x) ) ---> [ Low Volatility ]
[ Volatile Region ] -> ( Small Local ℓ(x) ) ---> [ High Volatility ]
For autonomous systems, this calibrated uncertainty is critical. Weizhe Chen et al. introduce the Attentive Kernel (AK) framework in Adaptive Robotic Information Gathering via Non-Stationary Gaussian Processes and POAM: Probabilistic Online Attentive Mapping for Efficient Robotic Information Gathering. By maximizing log marginal likelihood to transform standard stationary kernels into non-stationary representations, their approach provides highly accurate local uncertainty estimates. This guides robotic planners to bypass simple, flat terrains and prioritize sampling in regions with high spatial variation, maximizing the information gathered per sample.
Key Foundational Work
Computing with Infinite Networks (1996) — contributed 2 concept elements to this direction.
Mixtures of Gaussian Processes (2000) — contributed 1 concept elements to this direction.