The Gaussian Process Density Sampler

Ryan Prescott Adams, Iain Murray, David J. C. MacKayNeurIPS 2008 · OpenReview

Introduction

In Bayesian statistics, estimating a probability density function (PDF) from data is a classic challenge. Researchers often seek a model that is flexible (nonparametric) but also mathematically sound. The Gaussian Process Density Sampler (GPDS) is a novel framework for nonparametric Bayesian density estimation that leverages Gaussian Processes (GPs) without the usual computational headaches.

By framing density estimation as a rejection sampling process with a "latent history," the GPDS provides an exact, exchangeable way to infer unknown probability distributions without needing to numerically approximate intractable normalization constants.

The Normalization Bottleneck

One intuitive approach to density estimation is to use a Gaussian Process to define the "shape" of the density. However, a standard GP can take any real value, while a valid PDF must be strictly non-negative and integrate to one.

To fix this, researchers often use a "Logistic GP," squashing the GP through a sigmoid function. The catch is that the resulting normalization constant (the denominator that ensures the area under the curve is exactly 1) becomes a functional of the GP itself. This makes exact inference nearly impossible, forcing researchers to rely on restrictive approximations like grids or finite basis functions that scale poorly to higher dimensions.

The GPDS Approach: Latent Histories

The authors introduce the GPDS to bypass this bottleneck. The core idea is to define the density $f(x)$ as:

$$f(x) = \frac{1}{Z} \Phi(g(x)) \pi(x)$$

Where $\pi(x)$ is a simple base measure (like a Gaussian), $\Phi$ is a squashing function (like the logistic function), and $g(x)$ is a GP.

The breakthrough lies in how the authors handle the normalization constant $Z$. Instead of calculating it directly, they view the data as being generated via rejection sampling. In this view, points are proposed from the base measure $\pi(x)$ and "accepted" based on the value of the squashed GP.

To perform inference on observed data, the authors use a "retrospective sampling" trick. Since we don't know which points were rejected during the creation of our dataset, they treat the number and locations of these rejected points as latent variables. They then use Markov chain Monte Carlo (MCMC) to sample these "latent histories." By simulating the points that could have been rejected, they bypass the need to ever explicitly calculate the normalization constant $Z$.

This leads to a great intuition for density modeling, framed by the authors as the "Tent" Analogy: putting up a density model is like pitching a tent. The observed data points are like poles pushing the canvas up, but the "rejections" are like pegs pinning the canvas down. To accurately define a density, knowing where the probability is low is just as important as knowing where it is high.

Experiments and Results

The paper evaluates the GPDS on two primary tasks to test its flexibility and scalability:

  1. Toy Ring Data: A synthetic 2D dataset consisting of 100 points drawn from a ring with added Gaussian noise. This tests the model's ability to capture non-convex, non-linear shapes.
  2. Macaque Skull Reconstruction: A real-world, 10-dimensional dataset involving linear distances between anatomical landmarks on 228 rhesus macaque (Macaca mulatta) skulls, used to model morphological variations.
Density estimates on 2D data demonstrating the effect of varying GP hyperparameters.

On both the toy ring and the macaque skull data, the GPDS significantly outperformed standard Parzen windows (Kernel Density Estimation) in terms of predictive log probability. Furthermore, it performed comparably to other state-of-the-art Bayesian nonparametric methods, such as Infinite Mixtures of Gaussians (iMoG) and Dirichlet Diffusion Trees (DFT). While DFT was slightly better on the toy ring, all three Bayesian methods were roughly equal on the complex 10D skull data.

Crucially, unlike many other density estimators, the GPDS allows for the direct inference of GP hyperparameters (like length scales). This means the model can automatically learn the "smoothness" of the density directly from the data, as visualized in the varying contour shapes above.

Limitations and Future Work

While the GPDS offers an elegant, exact solution to a notoriously difficult problem, it does come with trade-offs:

  • Computational Cost: The primary bottleneck is the $O((N+M)^3)$ complexity of GP marginal likelihood evaluations, where $N$ is the number of data points and $M$ is the number of latent rejections. If the base measure $\pi(x)$ is a poor fit for the data, $M$ can become very large, making the algorithm slow.
  • MCMC Mixing: The "latent history" approach is mathematically exact, but basic Metropolis-Hastings steps can be slow to converge. The authors suggest that more sophisticated MCMC techniques or "sparse" GP approximations could be used to scale the method to much larger datasets.
  • Base Measure Sensitivity: The efficiency of the sampler depends heavily on the choice of the base measure $\pi(x)$, which acts as the proposal distribution for the rejection sampling.

Summary updated

Abstract

We present the Gaussian Process Density Sampler (GPDS), an exchangeable generative model for use in nonparametric Bayesian density estimation. Samples drawn from the GPDS are consistent with exact, independent samples from a fixed density function that is a transformation of a function drawn from a Gaussian process prior. Our formulation allows us to infer an unknown density from data using Markov chain Monte Carlo, which gives samples from the posterior distribution over density functions and from the predictive distribution on data space. We can also infer the hyperparameters of the Gaussian process. We compare this density modeling technique to several existing techniques on a toy problem and a skull-reconstruction task.
BibTeX
@inproceedings{adams2008gaussian,
  author = {Ryan Prescott Adams and Iain Murray and David J. C. MacKay},
  title = {{The Gaussian Process Density Sampler}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2008},
  url = {https://openreview.net/forum?id=BylqqO-OZB},
}
Figure