Regression with Input-dependent Noise: A Gaussian Process Treatment

Paul W. Goldberg, Christopher K. I. Williams, Christopher M. BishopNeurIPS 1997

Introduction

Standard Gaussian Processes (GPs) typically assume homoscedasticity—the premise that the noise or error in measurements remains constant across all input values. However, real-world data is rarely that well-behaved. Many systems exhibit heteroscedasticity, where the noise level depends heavily on the input ($x$). For instance, a sensor might become less reliable at extreme temperatures, or financial volatility might spike under specific market conditions.

While neural networks have been adapted to handle varying noise levels, integrating this capability into the elegant, non-parametric framework of Gaussian Processes has historically been difficult due to mathematical intractability. This paper introduces a "two-GP" framework to solve this: one Gaussian Process models the underlying signal, while a second GP models the log-variance of the noise, allowing the model to adapt to regions with naturally higher volatility.

A Two-GP Hierarchical Model

To capture both the signal and the varying noise, the authors propose a hierarchical model utilizing two distinct Gaussian Processes:

  1. The $y$-process: A standard GP used to predict the actual function values (the underlying signal).
  2. The $z$-process: A secondary GP that models the log of the noise variance. Modeling the log-variance ensures that the resulting noise estimate remains strictly positive.
Graphical model illustrating the two-GP framework, where the y-process models the signal and the z-process models the noise.

Because these two processes are coupled, the standard closed-form linear algebra typically used for GPs cannot be applied. Instead, the authors employ Markov Chain Monte Carlo (MCMC) methods to perform inference:

  • Gibbs Sampling: The algorithm alternates between fitting the signal curve (the $y$-process) and fitting the noise levels (the $z$-process).
  • Metropolis Algorithm: Used to update the hyperparameters, such as the length-scales that determine the smoothness of both the signal and the noise functions.
  • Rejection Sampling: Specifically applied to sample the noise levels at each individual data point.

A key insight of this approach is treating the noise level not merely as a nuisance parameter, but as a hidden (latent) function deserving its own GP prior. By assuming the noise changes smoothly, a GP becomes the natural tool to model it without requiring the user to guess a parametric shape for the noise function.

Experiments and Key Results

The paper primarily validates the model using a synthetic 1D dataset designed to exhibit clear heteroscedasticity. The underlying function is a sine wave ($y = 2 \sin(2\pi x)$), corrupted by independent Gaussian noise with a standard deviation that increases linearly from 0.5 at $x=0$ to 1.5 at $x=1$. The dataset consists of 60 points.

Synthetic 1D dataset showing a sine wave with noise variance increasing from left to right, along with the model's fitted mean.

The two-GP model successfully decoupled the signal from the noise in this synthetic test:

  • Accurate Noise Tracking: The inferred standard deviation closely matched the true linear increase used to generate the data.
  • Signal Recovery: The model correctly identified the length-scale of the underlying sine wave ($0.22 \pm 0.1$). This demonstrates that accurately modeling the noise prevents the primary signal-GP from being "confused" or overly smoothed by regions of high volatility.
  • Predictive Confidence: By accounting for input-dependent noise, the model generates meaningful predictive error bars that dynamically widen in high-noise regions and shrink in low-noise regions.

During training, the authors also noted that adding a relatively large "jitter" term (a small constant added to the diagonal of the covariance matrix) to the noise GP significantly improved the MCMC sampler's convergence.

Limitations and Future Work

While the two-GP framework provides a robust, non-parametric solution to heteroscedasticity, it comes with notable trade-offs:

  • Computational Intensity: Relying on MCMC rather than closed-form solutions makes this approach significantly slower than standard GP regression. Generating thousands of sampling iterations is computationally expensive.
  • Scalability: Standard GPs already scale poorly ($O(N^3)$), and layering MCMC on top exacerbates this issue. The primary validation used only 60 data points, making large-scale applications a significant challenge.
  • Hyperparameter Sensitivity: Although the model learns its own parameters, MCMC chains require careful tuning—such as setting appropriate "burn-in" periods—to ensure they converge to the correct posterior distribution.

Summary updated

BibTeX
@inproceedings{goldberg1997regression,
  author = {Paul W. Goldberg and Christopher K. I. Williams and Christopher M. Bishop},
  title = {{Regression with Input-dependent Noise: A Gaussian Process Treatment}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1997},
}
Figure