Deep convolutional Gaussian process

Kenneth Blomqvist, Samuel Kaski, Markus Heinonen2018

TL;DR

This paper introduces Deep Convolutional Gaussian Processes (DeepCGP), a novel Bayesian framework that combines the hierarchical feature extraction of Convolutional Neural Networks (CNNs) with the probabilistic modeling power of Gaussian Processes (GPs). This new architecture significantly improves image classification performance on standard benchmarks like MNIST and CIFAR-10, achieving state-of-the-art results for GP-based methods.

What Problem Does This Paper Address?

Traditional Gaussian Processes (GPs), while flexible and providing uncertainty estimates, often underperform in complex tasks like image classification due to their difficulty in modeling non-stationary patterns and long-range interactions. Shallow GPs struggle with these limitations. While deep GPs offer more capacity by stacking GP layers, they can suffer from degenerate covariances. Convolutional Neural Networks (CNNs), on the other hand, excel at image recognition by encoding hierarchical, translation-invariant feature extraction. However, CNNs typically require large amounts of data, can be difficult to optimize, and lack reliable uncertainty estimates. This paper aims to bridge this gap by developing a deep, convolutional GP architecture that leverages the strengths of both approaches, offering improved performance and probabilistic modeling for image classification.

Methods

The core innovation is the Deep Convolutional Gaussian Process (DeepCGP). This model constructs a deep architecture by stacking multiple convolutional GP layers. Each layer takes an input representation (initially the image) and applies a set of independent GPs to patches of this representation. These GPs act as patch response functions, producing a new, higher-level representation. This process is repeated for several layers, progressively extracting more complex features.

Key methodological aspects include:

  • Convolutional GP Layers: These layers apply GPs to local image patches. The model uses shared inducing points and kernels within each layer to manage computational complexity.
  • Probabilistic Feature Extraction: Unlike deterministic CNNs, each convolutional GP layer produces a probabilistic representation, allowing for uncertainty to propagate through the network.
  • Variational Inference: To handle the computational intractability of exact inference in deep GPs, the authors employ doubly stochastic variational inference. This approach uses Monte Carlo approximations for the expected likelihood and optimizes the evidence lower bound (ELBO) using stochastic gradient descent.
  • Final Classification Layer: The output of the last convolutional GP layer is fed into a GP classifier with a weighted convolutional kernel, similar to prior work by Van der Wilk et al. (2017).

Datasets

The paper evaluates the DeepCGP model on two standard image classification benchmarks:

  • MNIST: A dataset of 28x28 grayscale images of handwritten digits.
  • CIFAR-10: A dataset of 32x32 RGB color images of 10 object classes.

The authors preprocess the images by centering them and normalizing variance along color channels.

Key Insights and Ideas

  • Bridging the Gap: The DeepCGP successfully combines the hierarchical feature learning of CNNs with the principled probabilistic modeling of GPs, offering a powerful alternative for image classification.
  • Improved GP Performance: By introducing convolutional structure into deep GPs, the model significantly boosts performance on challenging image datasets, surpassing previous GP-based methods.
  • Hierarchical Feature Learning: The convolutional GP layers learn to extract increasingly abstract features at deeper levels, moving from simple edge detection to more complex pattern correlations.
  • No Rank Degeneracy: Despite being a deep GP model, the authors report that their architecture does not suffer from the rank degeneracy issues often encountered in deep GP compositions, possibly due to the multiple channel mappings and convolutional structure.

Key Results

The DeepCGP model achieves significant improvements in classification accuracy:

  • CIFAR-10: The model improves accuracy from 58.65% (shallow convolutional GP) to 75.89% with a 3-layer DeepCGP, a gain of over 17 percentage points. This also surpasses the hybrid neural-GP approach of Wilson et al. (2016a) by a notable margin.
  • MNIST: The model achieves an error rate of 0.56% with the three-layer DeepCGP, a substantial improvement over prior GP methods.

The paper also visualizes the learned representations and inducing points, showing how the model progressively extracts meaningful features.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Computational Cost: Deep GPs, even with variational inference, are computationally intensive. The authors were limited in experimenting with models deeper than three layers or with a stride of 1 in the first layer due to these costs.
  • Model Depth and Stride: Exploring models with more than three layers and investigating the impact of different stride values in the initial layers could further improve performance.
  • Efficiency Improvements: The authors suggest exploring techniques like Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) and kernel interpolation methods to enhance efficiency and speed up inference and prediction.
  • Rank Preservation: While their model avoids rank degeneracy, further investigation into the mechanisms behind this and potential improvements in rank-preserving deep GP models is warranted.

Summary updated

BibTeX
@misc{blomqvist2018deep,
  author = {Kenneth Blomqvist and Samuel Kaski and Markus Heinonen},
  title = {{Deep convolutional Gaussian process}},
  year = {2018},
}
Figure