Variational Mixtures of Gaussian Processes for Classification

Chen Luo, Shiliang SunIJCAI 2017 · OpenReview

Abstract

Gaussian Processes (GPs) are powerful tools for machine learning which have been applied to both classification and regression. The mixture models of GPs were later proposed to further improve GPs for data modeling. However, these models are formulated for regression problems. In this work, we propose a new Mixture of Gaussian Processes for Classification (MGPC). Instead of the Gaussian likelihood for regression, MGPC employs the logistic function as likelihood to obtain the class probabilities, which is suitable for classification problems. The posterior distribution of latent variables is approximated through variational inference. The hyperparameters are optimized through the variational EM method and a greedy algorithm. Experiments are performed on multiple real-world datasets which show improvements over five widely used methods on predictive performance. The results also indicate that for classification MGPC is significantly better than the regression model with mixtures of GPs, different from the existing consensus that their single model counterparts are comparable.

Introduction

Gaussian Processes (GPs) are considered a "gold standard" for probabilistic modeling due to their flexibility and principled uncertainty quantification. However, they face two significant challenges: scalability and multi-modality. The computational cost of GPs scales cubically ($O(N^3)$) with the number of data points, limiting their use on large datasets. Furthermore, a single GP assumes a stationary global structure, which often fails to capture complex data with varying local behaviors or multiple "modes."

While "Mixture of Experts" models have successfully addressed these issues in GP regression, extending them to classification is non-trivial. Regression benefits from Gaussian likelihoods that allow for closed-form solutions. In contrast, classification requires non-Gaussian likelihoods (e.g., logistic functions), making the posterior distribution intractable and traditionally necessitating slow Markov Chain Monte Carlo (MCMC) methods. This paper introduces MGPC (Mixture of Gaussian Processes for Classification), a generative framework that utilizes variational inference to provide a scalable and effective mixture model for classification.

Methodology

The MGPC framework is built upon several key technical innovations designed to maintain flexibility while ensuring computational tractability.

The graphical model representation of the proposed MGPC framework, illustrating the dependencies between latent variables, inputs, and class labels.

The model employs a "linear GP" representation, introducing intermediate latent variables to break dependencies between outputs and facilitate mean-field variational inference. Unlike regression-based mixtures, MGPC uses a logistic likelihood to map latent variables directly to class probabilities. To handle the mixture component, a Dirichlet Process (DP) prior is used via a stick-breaking construction, theoretically allowing for an infinite number of experts.

For optimization, the authors use a Variational EM algorithm to find a lower bound on the log-likelihood. A greedy algorithm is employed to select "support sets"—specific subsets of data points that each GP expert focuses on—maximizing the model's evidence while reducing the overall computational complexity from $O(N^3)$ to the sum of smaller cubic operations.

Experimental Setup

The performance of MGPC was evaluated using nine real-world datasets from the UCI Machine Learning Repository, covering medical, biological, and social domains. These datasets, including Blood, Fertility, Haberman, and others, vary in dimensionality (3 to 22 features) and instance counts, providing a comprehensive benchmark for general classification performance.

Results and Discussion

The experimental results demonstrate that MGPC consistently outperforms standard baselines. As shown in the table below, MGPC achieved the highest classification accuracy on seven out of the nine tested datasets.

Comparison of classification accuracy (mean ± standard deviation) across nine datasets for MGPC and five baseline methods.

Key insights from the results include:

  • Classification vs. Regression Proxies: MGPC significantly outperforms Mixture of GP Regression (MGPR) models that use a sign-based proxy for classification. This highlights the importance of explicitly modeling the classification likelihood.
  • Efficiency through Decomposition: By dividing the training set into smaller support sets, MGPC achieves better scalability than single GP models without sacrificing predictive power.
  • Uncertainty Quantification: Beyond accuracy, MGPC showed superior predictive log-likelihood values, indicating more robust and "confident" probability estimations compared to regression-based alternatives.

Conclusion

MGPC successfully brings the benefits of mixture modeling to Gaussian Process classification. By combining a logistic likelihood with variational inference and a Dirichlet Process prior, the model addresses both the scalability and multi-modality limitations of standard GPs. While the current implementation relies on a greedy selection for support sets and a fixed truncation for the number of experts, future work may explore more adaptive methods for automatically inferring these parameters. Overall, MGPC represents a significant step forward in probabilistic classification for complex, multi-modal datasets.

Summary updated

Abstract

Gaussian Processes (GPs) are powerful tools for machine learning which have been applied to both classification and regression. The mixture models of GPs were later proposed to further improve GPs for data modeling. However, these models are formulated for regression problems. In this work, we propose a new Mixture of Gaussian Processes for Classification (MGPC). Instead of the Gaussian likelihood for regression, MGPC employs the logistic function as likelihood to obtain the class probabilities, which is suitable for classification problems. The posterior distribution of latent variables is approximated through variational inference. The hyperparameters are optimized through the variational EM method and a greedy algorithm. Experiments are performed on multiple real-world datasets which show improvements over five widely used methods on predictive performance. The results also indicate that for classification MGPC is significantly better than the regression model with mixtures of GPs, different from the existing consensus that their single model counterparts are comparable.
BibTeX
@inproceedings{luo2017variational,
  author = {Chen Luo and Shiliang Sun},
  title = {{Variational Mixtures of Gaussian Processes for Classification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  url = {https://openreview.net/forum?id=Hy-8RQzObS},
}
Figure