Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes

Ruslan Salakhutdinov, Geoffrey E. HintonNeurIPS 2007 · OpenReview

This paper, authored by machine learning pioneers Ruslan Salakhutdinov and Geoffrey Hinton, represents a landmark intersection between the then-emerging field of Deep Learning and the established framework of Bayesian non-parametrics.

TL;DR

The authors propose a hybrid approach that uses Deep Belief Nets (DBNs) to automatically learn a non-linear feature mapping for Gaussian Processes (GPs). By pre-training on unlabeled data and fine-tuning with labels, they create a "smart" covariance kernel that significantly outperforms standard kernels on high-dimensional, structured data like images and text.

Introduction

Gaussian Processes (GPs) are powerful tools for regression and classification because they provide well-calibrated uncertainty estimates. However, they suffer from a "representation" problem. A GP’s performance depends entirely on its covariance kernel, which defines how "similar" two data points are.

Standard kernels (like the RBF or Gaussian kernel) struggle with high-dimensional, highly structured data. For example, in a raw pixel space, two images of the same face might look mathematically "distant" if one is slightly rotated. Furthermore, GPs are traditionally discriminative models that cannot easily leverage the vast amounts of unlabeled data often available in real-world scenarios. The authors aim to bridge this gap by using deep learning to "discover" the right similarity metric.

Methodology

The authors introduce a three-stage pipeline to construct a more effective kernel:

  1. Unsupervised Pre-training: They use a stack of Restricted Boltzmann Machines (RBMs) to learn a Deep Belief Net (DBN) on a large set of unlabeled data. This stage uses a fast, greedy learning algorithm to capture high-order correlations and "understand" the underlying structure of the data (e.g., what makes a face look like a face).
  2. Feature Mapping: The learned DBN acts as a non-linear mapping function, $F(\mathbf{x}|\mathbf{W})$, which transforms high-dimensional input $\mathbf{x}$ into a lower-dimensional feature space $\mathbf{Z}$.
DBN-GP Architecture
  1. The DBN-GP Kernel: Instead of applying a Gaussian kernel to the raw input, they apply it to the top-layer features of the DBN: $K(\mathbf{x}_i, \mathbf{x}_j) = \exp(-\|F(\mathbf{x}_i|\mathbf{W}) - F(\mathbf{x}_j|\mathbf{W})\|^2)$.

  2. Discriminative Fine-tuning: Finally, they use backpropagation to adjust the millions of parameters ($\mathbf{W}$) in the DBN. Unlike standard deep learning, the objective here is to maximize the marginal log-likelihood of the Gaussian Process, ensuring the feature space is optimized specifically for the prediction task.

Datasets

The researchers evaluated their method on three distinct types of structured data:

  • Olivetti Faces: 13,000 images (28x28) of faces. The task was to extract face orientation. They also tested a "difficult" version with random rectangular occlusions to test robustness.
Occluded Faces Dataset
  • MNIST Digits: 60,000 images of handwritten digits. Tasks included magnitude regression (predicting the digit value) and odd vs. even classification.
  • Reuters Corpus Volume II: Over 800,000 newswire stories, represented as word-count vectors (2,000 dimensions). The task was to classify stories into major categories.

Key Insights

  • Unlabeled Data as a Prior: The paper demonstrates that modeling the distribution of the input data $p(\mathbf{X})$ provides a massive head start. The DBN "initializes" the kernel in a way that makes the subsequent supervised learning much more efficient.
  • Feature Space vs. Input Space: Standard kernels are "blind" to structure. By moving the kernel to the top of a DBN, the GP operates in a space where semantic similarities (like the angle of a head) are explicit, while irrelevant noise (like individual pixel variations) is filtered out.
  • Efficiency in Small-Label Regimes: One of the most striking findings is that a DBN-GP trained on just 100 labeled examples can outperform a standard GP trained on 1,000 examples. This makes the method highly valuable for domains where labeling data is expensive (e.g., medical imaging).

Results

  • Face Orientation: On the non-occluded face data, the fine-tuned DBN-GP achieved a Root Mean Squared Error (RMSE) of 6.31, compared to 16.36 for a standard GP.
  • Robustness: When faces were partially blocked (occluded), the DBN-GP remained highly effective (RMSE 9.23), while the standard GP's performance degraded significantly (RMSE 17.98).
  • Text Classification: On the Reuters dataset, the DBN-GP achieved an error rate (1-AUROC) of 0.0458, significantly better than the standard GP's 0.0645.

Limitations and Future Work

  • The $O(N^3)$ Bottleneck: While the DBN handles high-dimensional features efficiently, the Gaussian Process still requires inverting a covariance matrix, which scales cubically with the number of labeled training cases ($N$). This limits the method to a few thousand labeled points unless sparse GP approximations are used.
  • Complexity: Training a DBN and then fine-tuning it through a GP objective is computationally intensive and requires careful hyperparameter tuning compared to a simple SVM or standard GP.
  • Future Directions: The authors suggest that this approach could be extended to other kernel-based methods, such as Support Vector Machines (SVMs), and that future research should compare this to other semi-supervised learning algorithms.

Summary updated

Abstract

We show how to use unlabeled data and a deep belief net (DBN) to learn a good covariance kernel for a Gaussian process. We first learn a deep generative model of the unlabeled data using the fast, greedy algorithm introduced by Hinton et.al. If the data is high-dimensional and highly-structured, a Gaussian kernel applied to the top layer of features in the DBN works much better than a similar kernel applied to the raw input. Performance at both regression and classification can then be further improved by using backpropagation through the DBN to discriminatively fine-tune the covariance kernel.
BibTeX
@inproceedings{salakhutdinov2007using,
  author = {Ruslan Salakhutdinov and Geoffrey E. Hinton},
  title = {{Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2007},
  url = {https://openreview.net/forum?id=ryWJi8Z_ZS},
}
Figure