Hierarchical Gaussian Processes for Nonstationary Regression

This direction is merged into another direction: Gaussian Process scalability. Its last version stays readable and citable here.

Version 1 of 122 papers30 conceptsfrozen 2026-10-04

Standard Gaussian Processes (GPs) are often described as the "Swiss Army Knife" of Bayesian regression. They provide elegant, non-parametric predictions with built-in uncertainty. However, they typically rely on a critical, often hidden assumption: stationarity. In a stationary GP, the statistical properties of the signal—its "wiggliness" (lengthscale), its amplitude (signal variance), and its noise level—are assumed to be the same everywhere.

In the real world, data rarely plays by these rules. A sensor monitoring environmental changes might encounter calm periods followed by sudden, volatile shifts. An animal's movement might transition from slow grazing to rapid flight. To model these phenomena, researchers have moved toward hierarchical non-stationary models, where the parameters governing the GP are themselves treated as stochastic processes.

The Problem with Global Smoothness

In a traditional GP, the kernel (or covariance function) determines how much influence one observation has on another based on their distance. If you set a single global lengthscale $\ell$, you are forcing the model to assume that the function's rate of change is uniform.

As Christopher J. Paciorek and Mark J. Schervish pointed out in Nonstationary Covariance Functions for Gaussian Process Regression, this fails on complex data. If the lengthscale is too large, the model oversmooths rapid transitions; if it is too small, it overfits stable regions. Their solution was to introduce a class of non-stationary kernels where the local correlation structure changes smoothly across the input space.

Hyperparameters as Latent Functions

The core intuition of this research direction is to stop treating hyperparameters as fixed constants and start treating them as latent functions of the input $x$.

In the framework proposed by Markus Heinonen and Harri Lähdesmäki in Non-Stationary Gaussian Process Regression with Hamiltonian Monte Carlo, three distinct pillars of the GP are modeled as input-dependent:

  1. Lengthscale $\ell(x)$: Controls local smoothness.
  2. Signal Variance $\sigma_f^2(x)$: Controls local amplitude.
  3. Noise Variance $\sigma_n^2(x)$: Controls local heteroscedasticity (varying noise).

By placing secondary GP priors on the logarithms of these parameters, the model ensures they remain positive while allowing them to evolve gracefully. This hierarchical structure allows the model to "learn" that one region of the data is volatile while another is smooth, effectively disentangling structural changes from random noise.

Building Intuition: The Expert Analogy

One way to visualize this is through the lens of "local experts." In Embarrassingly Parallel Inference for Gaussian Processes, Michael Minyi Zhang discusses partitioning models into local experts with independent hyperparameters.

Similarly, Martin Trapp and colleagues use Sum-Product Networks to create a mixture of GP experts. Each expert specializes in a specific region of the input space. The hierarchical approach essentially creates an infinite, continuous field of these experts. Instead of hard boundaries between regions, the latent GPs provide a smooth transition, allowing the model to adapt its "reach" and "sensitivity" dynamically.

Overcoming Computational Hurdles

The primary reason stationary GPs remained the standard for so long is computational. Modeling hyperparameters as latent GPs makes the marginal likelihood non-Gaussian and the inference problem significantly harder.

Recent work has focused on making these hierarchical models scalable:

  • Sampling-based methods: Markus Heinonen utilized Hamiltonian Monte Carlo (HMC) with "whitening" transformations to navigate the complex posterior distributions of these models.
  • Variational Inference: Dirk Husmeier and Ionut Paun introduced stochastic variational inference (SVI) for non-stationary GPs, enabling them to handle datasets with millions of points by using sparse GP approximations for the latent parameter processes.
  • Inducing Points: Kevin Murphy and Zeel B. Patel demonstrated that inducing point approximations can efficiently disentangle epistemic uncertainty (what the model doesn't know) from aleatoric noise (inherent data variability) in non-stationary settings.

Applications in the Wild

This flexibility is not just a theoretical luxury; it is a necessity in specialized fields. In ecology, Colin J. Torney and Juan M. Morales apply these models to animal movement data, where the correlation lengthscale represents the "memory" or persistence of a path—a trait that changes as an animal switches from foraging to migrating.

In environmental monitoring, Sahil Garg and Fabio Ramos developed NOSTILL-GP to capture space-time phenomena where the influence of a sensor reading varies depending on local terrain or weather conditions. By allowing the model to learn its own smoothness, these researchers can reconstruct complex signals that stationary models would simply dismiss as noise.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_gaussian_processes_for_nons_7b090366,
  title = {Hierarchical Gaussian Processes for Nonstationary Regression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-gaussian-processes-for-nonstationary-regression/rdr_7b090366eeb54a939c3c89df88db61fe},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_hierarchical_gaussian_processes_for_nons_7b090366_v1,
  title = {Hierarchical Gaussian Processes for Nonstationary Regression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-gaussian-processes-for-nonstationary-regression/rdr_7b090366eeb54a939c3c89df88db61fe/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}