Mode Collapse in Variational Deep Gaussian Processes

Francisco Javier Sáez-Maldonado, Juan Maroñas, Daniel Hernández-LobatoBDU @ NeurIPS 2024 · OpenReview

Abstract

Deep Gaussian Processes (DGPs) define a hierarchical model capable of learning complex, non-stationary processes. Exact inference is intractable in DGPs, so a variational distribution is used in each layer. One of the main challenges when training DGPs is the prevention of a phenomenon known as mode collapse where, during training, the variational distribution becomes the prior distribution which is a minimizer of the KL-Divergence term in the ELBO. There are two main factors that influence the optimization process: the mean function of the inner GPs and the usage of the whitened representation of the variational distribution. In this work, we propose a data-driven initialization of the variational parameters that a) at initialization, predicts an already good approximation of the objective function, b) avoids mode collapse c) is supported by a theoretical analysis of the behavior of the KL divergence and experimental results in real-world datasets.
BibTeX
@inproceedings{saezmaldonado2024mode,
  author = {Francisco Javier Sáez-Maldonado and Juan Maroñas and Daniel Hernández-Lobato},
  title = {{Mode Collapse in Variational Deep Gaussian Processes}},
  booktitle = {BDU @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=AyJweU0IJg},
}

Loading related content...

Figure