Hybrid Discriminative-Generative Approach with Gaussian Processes

Ricardo Andrade Pacheco, James Hensman, Max Zwiessele, Neil D. LawrenceAISTATS 2014 · OpenReview

Abstract: Machine learning practitioners are often faced with a choice between a discriminative and a generative approach to modelling. Here, we present a model based on a hybrid approach that breaks down some of the traditional barriers between these two paradigms. By combining Expectation Propagation (EP) with variational sparse Gaussian Process (GP) techniques, we introduce a unified framework capable of handling mixed discrete-continuous data, missing or uncertain inputs, and label-informed manifold learning.

The Discriminative-Generative Divide

In traditional machine learning, practitioners typically choose between two distinct paths. Discriminative models are highly effective at predicting a target $y$ from inputs $X$, but they generally assume $X$ is perfectly known. When inputs are missing or noisy, these models often fail or produce overconfident, incorrect predictions. Conversely, generative models model the joint distribution of $X$ and $y$. While they handle missing data naturally, they are often less optimized for specific classification tasks.

Gaussian Processes (GPs) have long been a staple of discriminative modeling. However, extending them to handle non-Gaussian data (such as binary labels) while simultaneously accounting for uncertain inputs or performing unsupervised dimensionality reduction has remained mathematically challenging. This research proposes a "marriage" of techniques to bridge this gap in a single, tractable framework.

The EP-DTC Framework

The core innovation of this work is the combination of two established methods: Expectation Propagation (EP) and Variational Sparse GPs. EP is an iterative algorithm used to approximate complex, non-Gaussian likelihoods, such as the Bernoulli distribution used for binary classification. The authors integrate this into a variational sparse GP objective—specifically the Deterministic Training Conditional (DTC) variant—to create EP-DTC.

This algorithm allows for the approximation of the marginal likelihood even when data is non-Gaussian. Crucially, the framework extends to handle uncertain inputs. Instead of treating inputs $X$ as fixed points, the model treats them as distributions. By propagating this uncertainty through the GP covariance function, the model adjusts its confidence based on the reliability of the input data.

Handling Uncertain Inputs

Decision boundary and uncertainty region when input variance is high (var(x)=0.1).

As shown in the figure above, when input uncertainty is high, the model's decision boundary becomes more conservative, and the uncertainty regions expand. This prevents the model from making overconfident predictions in regions where the input data is poorly defined.

Robustness and "Graceful Failure"

A key insight of this hybrid approach is the concept of "graceful failure." In experiments using the Olivetti Faces dataset, the authors trained a classifier to detect whether a subject was wearing glasses. By systematically removing pixels from test images, they compared the hybrid model to a standard GP.

While a standard GP remained "certain" that a person was not wearing glasses even as data disappeared (assigning a probability of 0), the hybrid model correctly became less certain. As more data was removed, the hybrid model's prediction reverted toward the dataset's prior probability (~30%). Essentially, the model "knows what it doesn't know," providing a more robust measure of uncertainty in the face of missing information.

Dimensionality Reduction on Mixed Data

The framework also excels at manifold learning, particularly when dealing with mixed data types. By treating class labels as an additional "view" of the data within a latent variable model, the resulting visualizations are better organized than those produced by purely unsupervised methods.

3D latent space visualization of the Zoo dataset, clustering animals into biological categories.

In the Zoo dataset experiment—which consists of 101 animals described by 15 boolean attributes and one numerical attribute—the model successfully clustered the animals into biological categories (mammals, birds, fish, etc.) within a 3D latent space. This was achieved by using Bernoulli likelihoods for boolean traits and Gaussian likelihoods for numerical ones, all within the same unified model.

Conclusion

The hybrid EP-DTC framework demonstrates that practitioners do not have to sacrifice discriminative accuracy for generative flexibility. On standard classification benchmarks, the model performs competitively with state-of-the-art methods like EP-FITC while offering significantly better robustness to missing data.

Future work points toward the integration of Deep Gaussian Processes to capture even more complex hierarchies, as well as exploring "Tilted Variational Bayes" (TVB) as a potential alternative to EP for providing even more rigorous bounds on non-Gaussian likelihoods.

Summary updated

Abstract

Machine learning practitioners are often faced with a choice between a discriminative and a generative approach to modelling. Here, we present a model based on a hybrid approach that breaks down so...
BibTeX
@inproceedings{pacheco2014hybrid,
  author = {Ricardo Andrade Pacheco and James Hensman and Max Zwiessele and Neil D. Lawrence},
  title = {{Hybrid Discriminative-Generative Approach with Gaussian Processes}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2014},
  url = {https://openreview.net/forum?id=_eM4TsUaq_},
}
Figure