An Alternative Infinite Mixture Of Gaussian Process Experts

Edward Meeds, Simon OsinderoNeurIPS 2005 · OpenReview

Introduction

Gaussian Processes (GPs) are highly regarded as the "Swiss Army knife" of Bayesian regression due to their flexibility and robust uncertainty quantification. However, standard GPs suffer from three major limitations: they scale poorly with large datasets ($O(n^3)$ complexity), they assume stationary noise and smoothness across the entire input space, and they are typically restricted to conditional mappings ($P(y|x)$). This conditional restriction makes it difficult to handle missing input data or perform inverse predictions.

To address these issues, this paper introduces the Alternative Infinite Mixture of Gaussian Process Experts (AiMoGPE). By modeling the joint distribution of both inputs and outputs—rather than just the conditional mapping—the authors create a flexible, generative framework capable of handling non-stationary data, multi-modality, and inverse functional mappings.

The Generative Approach: AiMoGPE

Most Mixture of Experts (MoE) models rely on a separate "gating network" to decide which expert handles a given input, focusing solely on predicting the output $y$. The AiMoGPE takes a fundamentally different, generative approach. It treats both the input $x$ and the output $y$ as being generated by the model.

Detailed graphical model of the AiMoGPE framework.

In this framework, each "expert" consists of two components:

  1. A Multivariate Gaussian: This describes the density and region of the input space ($x$) that the expert "owns."
  2. A Gaussian Process: This models the output mapping ($y$) specifically for the points within that expert's domain.

Instead of a dedicated gating network, gating happens naturally via posterior responsibility. If an input $x$ falls into the spatial region defined by an expert's Gaussian, that expert naturally takes the lead in predicting $y$. Furthermore, the model employs a Dirichlet Process (DP) prior, granting it infinite flexibility. The DP allows the model to automatically infer the optimal number of experts from the data complexity, rather than requiring the user to pre-specify a fixed number. Because exact inference is analytically intractable, the model relies on Markov Chain Monte Carlo (MCMC) methods to learn the parameters.

Key Capabilities and Insights

Moving to a full generative model unlocks several powerful capabilities:

  • Handling Missing Data: Because the model learns the underlying distribution of the inputs, it can make principled predictions even when input coordinates are partially missing.
  • Reverse-Conditioning: Since the model captures the joint probability $P(x, y)$, it can be queried in reverse to sample $P(x|y)$. The authors successfully demonstrated predicting likely input locations given a specific output value.
  • Divide and Conquer Scaling: By partitioning the dataset among multiple local experts, the model only needs to invert several small covariance matrices rather than one massive $n \times n$ matrix, offering a pathway to better scalability.

Experimental Results

The authors evaluated AiMoGPE on both synthetic and real-world datasets to test its handling of multi-modality and non-stationarity.

Multi-modality on Synthetic Data: A custom toy dataset was generated with four distinct, overlapping continuous functions featuring varying noise levels. A standard single GP fails here, attempting to average the overlapping functions and producing massive error bars. The AiMoGPE, however, successfully identified and separated the distinct functional "strands."

Model performance on the synthetic toy dataset, demonstrating its ability to separate overlapping functional strands into distinct experts.

Non-Stationary Noise on Real Data: The model was also tested on the classic Motorcycle Impact dataset, which records acceleration over time during a crash. This dataset is notoriously non-stationary, with noise levels spiking dramatically after the moment of impact. While a standard GP is forced to compromise with a single, mediocre noise estimate across the entire timeline, AiMoGPE accurately captured the transition by assigning different experts to the low-noise and high-noise regimes.

Limitations and Future Directions

While powerful, the AiMoGPE has a few limitations. Currently, each expert uses a single Gaussian to define its territory in the input space, which can be restrictive for complex input distributions. The authors suggest that future iterations could allow experts to use their own "mini-mixtures" of Gaussians. Additionally, the model might exhibit a "clustering bias," creating new experts simply because data points are spatially clustered, even if a single GP could model the underlying function. Finally, while the divide-and-conquer approach theoretically improves scaling, testing the MCMC sampler's limits on truly massive datasets remains an area for future research.

Summary updated

Abstract

We present an infinite mixture model in which each component com- prises a multivariate Gaussian distribution over an input space, and a Gaussian Process model over an output space. Our model is neatly able to deal with non-stationary covariance functions, discontinuities, multi- modality and overlapping output signals. The work is similar to that by Rasmussen and Ghahramani [1]; however, we use a full generative model over input and output space rather than just a conditional model. This al- lows us to deal with incomplete data, to perform inference over inverse functional mappings as well as for regression, and also leads to a more powerful and consistent Bayesian specification of the effective ‘gating network’ for the different experts.
BibTeX
@inproceedings{meeds2005alternative,
  author = {Edward Meeds and Simon Osindero},
  title = {{An Alternative Infinite Mixture Of Gaussian Process Experts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2005},
  url = {https://openreview.net/forum?id=HJV_hd-OWS},
}
Figure