Deep Latent-Variable Kernel Learning

Haitao Liu, Yew-Soon Ong, Xiaomo Jiang, Xiaofang WangIEEE Trans. Cybernetics 2020 · OpenReview

TL;DR

This paper introduces Deep Latent-Variable Kernel Learning (DLVKL), a novel hybrid model that combines the representation power of neural networks with the regularization capabilities of Gaussian Processes. DLVKL addresses the over-fitting issue in existing Deep Kernel Learning (DKL) methods, especially on small datasets, by employing stochastic encoding and a flexible prior, leading to improved performance and regularization.

What Problem Does This Paper Address?

Traditional Gaussian Processes (GPs) are powerful for their non-parametric regularization and uncertainty quantification but struggle with scalability and learning complex representations from massive datasets. Neural Networks (NNs) excel at learning rich representations but lack the inherent regularization and uncertainty estimation of GPs. Deep Kernel Learning (DKL) attempts to bridge this gap by using an NN as a deterministic encoder for a GP. However, the deterministic nature of the NN encoder in DKL can weaken the GP's regularization effect, particularly on small datasets, leading to over-fitting and under-estimated prediction variance. This paper aims to create a more robust and better-regularized hybrid model by introducing stochasticity into the encoding process and developing a more flexible prior.

Methods

The core contribution is the Deep Latent-Variable Kernel Learning (DLVKL) framework. Unlike DKL's deterministic encoder, DLVKL uses a stochastic encoding process for the latent variables, which helps regularize the representation learning. This is achieved by introducing a prior distribution over the latent variables that is conditioned on the input ($p(z|x)$). The DLVKL model then optimizes an Evidence Lower Bound (ELBO) that includes a KL divergence term between the approximate posterior of the latent variables and this prior, thus encouraging the latent representations to be well-behaved.

To further enhance DLVKL, the authors propose DLVKL-NSDE, which introduces two key improvements:

  1. Expressive Variational Posterior via Neural Stochastic Differential Equations (NSDE): Instead of a simple Gaussian posterior, DLVKL-NSDE uses NSDEs to model the evolution of latent variables over a continuous time period. This allows for more complex and expressive variational posteriors, better approximating the true posterior and reducing the gap to exact inference.
  2. Flexible, Hybrid Prior: The prior distribution $p(z|x)$ is made more flexible by combining an SDE-based prior with the variational posterior. This is controlled by a trade-off parameter $\beta$, allowing for adjustable regularization. A $\beta=1.0$ emphasizes regularization, while $\beta \to 0$ leans towards the DKL approach.

Datasets

The paper evaluates the proposed models on a variety of datasets:

  • Toy Cases: Two toy datasets are used to illustrate the model's behavior, including a regression case with a step behavior and a classification case.
  • Regression Datasets: Six UCI regression datasets are used: boston, concrete, wine-red, keggdirected, kin40k, and protein.
  • Classification Datasets: Two UCI classification datasets are used: connect-4 and higgs.
  • Image Classification Dataset: cifar-10.
  • Unsupervised Learning Dataset: mnist (handwritten digits).

Key Insights and Ideas

  • Deterministic Encoders Weaken Regularization: The paper highlights that the deterministic nature of NN encoders in DKL, while good for representation learning, can undermine the GP's regularization, especially on small datasets.
  • Stochastic Encoding for Regularization: Introducing stochasticity into the latent variable encoding process (DLVKL) is crucial for achieving better regularization and preventing over-fitting.
  • NSDE for Expressive Posteriors: Using Neural Stochastic Differential Equations (NSDE) allows for more sophisticated and accurate variational posteriors, which can better capture the true posterior distribution of latent variables.
  • Hybrid Prior for Tunable Regularization: A flexible prior that balances information from an SDE-based prior and the variational posterior (controlled by $\beta$) provides a mechanism for adjustable regularization, allowing the model to adapt to different dataset sizes and complexities.
  • Posterior Collapse is a Real Issue: The paper demonstrates that even with a GP, posterior collapse can occur, leading to a degenerate constant predictor. The proposed flexible prior helps mitigate this.

Key Results

  • Small Datasets: DLVKL-NSDE performs similarly to well-calibrated GPs (like DiffGP) on small datasets, indicating effective regularization.
  • Large Datasets: DLVKL-NSDE shows superiority over existing deep GP methods on larger datasets, leveraging the power of deep latent representations.
  • Regression Performance: DLVKL-NSDE achieves competitive RMSE and NLL scores, often matching or exceeding other methods, particularly on larger datasets.
  • Classification Performance: DLVKL-NSDE demonstrates strong classification accuracy and NLL results.
  • Unsupervised Learning: The DLVKL model (a special case of DLVKL-NSDE) shows promising results on the MNIST dataset for unsupervised learning, with reasonable digit reconstruction and latent space clustering.
  • Impact of $\beta$: The parameter $\beta$ effectively controls the trade-off between regularization and representation learning. A higher $\beta$ is beneficial for small datasets to prevent over-fitting, while a smaller $\beta$ is preferred for large datasets to exploit deep latent representations.

Limitations and Future Work

  • High-Dimensional Data with NSDE: The NSDE transformation requires latent variables to have the same dimensionality over time, making DLVKL-NSDE unsuitable for very high-dimensional data like images. DLVKL, however, does not have this limitation.
  • Prior Initialization for SDE: For high-dimensional inputs where $d_z < d_x$, simple PCA might be needed to obtain the prior mean for the SDE.
  • Computational Complexity: The use of SDEs and NSDEs can increase computational complexity, especially with longer flow times ($T$) and more steps ($L$). The authors suggest a trade-off is needed.
  • Future Work: The paper suggests that more advanced GP paradigms, such as convolutional GPs, could be explored for image data to further improve performance.

Summary updated

Abstract

Deep kernel learning (DKL) leverages the connection between Gaussian process (GP) and neural networks (NN) to build an end-to-end, hybrid model. It combines the capability of NN to learn rich representations under massive data and the non-parametric property of GP to achieve automatic regularization that incorporates a trade-off between model fit and model complexity. However, the deterministic encoder may weaken the model regularization of the following GP part, especially on small datasets, due to the free latent representation. We therefore present a complete deep latent-variable kernel learning (DLVKL) model wherein the latent variables perform stochastic encoding for regularized representation. We further enhance the DLVKL from two aspects: (i) the expressive variational posterior through neural stochastic differential equation (NSDE) to improve the approximation quality, and (ii) the hybrid prior taking knowledge from both the SDE prior and the posterior to arrive at a flexible trade-off. Intensive experiments imply that the DLVKL-NSDE performs similarly to the well calibrated GP on small datasets, and outperforms existing deep GPs on large datasets.
BibTeX
@article{liu2020deep,
  author = {Haitao Liu and Yew-Soon Ong and Xiaomo Jiang and Xiaofang Wang},
  title = {{Deep Latent-Variable Kernel Learning}},
  journal = {IEEE Trans. Cybernetics},
  year = {2020},
  url = {https://openreview.net/forum?id=HI1yhj0OJl},
}
Figure