Latent Gaussian process with composite likelihoods and numerical quadrature

Siddharth Ramchandran, Miika Koskinen, Harri Lähdesmäki

Abstract

Clinical patient records are an example of high-dimensional data that is typically collected from disparate sources and comprises of multiple likelihoods with noisy as well as missing values. In this work, we propose an unsupervised generative model that can learn a low-dimensional representation among the observations in a latent space, while making use of all available data in a heterogeneous data setting with missing values. We improve upon the existing Gaussian process latent variable model (GPLVM) by incorporating multiple likelihoods and deep neural network parameterised back-constraints to create a non-linear dimensionality reduction technique for heterogeneous data. In addition, we develop a variational inference method for our model that uses numerical quadrature. We establish the effectiveness of our model and compare against existing GPLVM methods on a standard benchmark dataset as well as on clinical data of Parkinson’s disease patients treated at the HUS Helsinki University Hospital.

Introduction

Dimensionality reduction is a cornerstone of unsupervised learning, helping researchers find low-dimensional patterns in high-dimensional data. While the Gaussian Process Latent Variable Model (GPLVM) is a powerful tool for this, it has a significant "real-world" weakness: it typically assumes that all data follows a Gaussian distribution (homogeneous data). In fields like medicine, data is rarely that clean. A single patient record might contain binary diagnosis codes, integer-based count data, and continuous laboratory measurements, often with many missing values. Standard GPLVMs struggle here because non-Gaussian likelihoods make the underlying mathematics (the "inference") intractable. Furthermore, standard GPLVMs don't always guarantee that data points close to each other in the original space remain close in the latent space—a property known as local distance preservation.

Methods

The proposed "shared GPLVM" framework addresses these gaps through three key technical innovations:

  1. Composite Likelihoods: Instead of forcing a single distribution on the entire dataset, the model assigns specific likelihoods (Gaussian, Binomial, Beta, Poisson, or Categorical) to different variables. These are linked to a shared latent space via Gaussian Processes and deterministic "link functions" (like the sigmoid or exponential functions).
Conceptual diagram of the shared GPLVM framework incorporating multiple likelihoods for heterogeneous data.
  1. Numerical Quadrature: To solve the "intractable math" problem caused by non-Gaussian likelihoods, the authors use Gauss-Hermite quadrature. This is a numerical technique to approximate complex integrals. They found this approach to be more computationally efficient and stable than traditional sampling-based methods.
  2. Variational Recognition Models (Back-constraints): Borrowing an idea from Variational Autoencoders (VAEs), they use deep neural networks to map the observed data back to the latent space. This acts as a "back-constraint," ensuring that similar observations are clustered together in the latent representation and allowing the model to scale to large datasets via mini-batching.

Results and Discussion

The model was evaluated on two primary datasets: clinical patient data from 1,400 Parkinson’s disease patients at HUS Helsinki University Hospital (comprising 46 variables across binomial, Gaussian, and beta distributions) and a simulated heterogeneous MNIST dataset.

On the clinical dataset, the composite likelihood approach achieved a significantly higher predictive log-likelihood compared to standard GPLVMs that assumed all data was Gaussian. The model successfully identified distinct patient subtypes. By analyzing the latent clusters, the researchers found groups with contrasting clinical characteristics (e.g., different laboratory profiles), which is a vital step toward personalized medicine.

Clustering results on the Parkinson's disease dataset, showing cluster counts (a), consensus index (b), and the latent space representation (c).

The use of consensus clustering and robustness analysis confirmed that the identified patient sub-groups were stable and not just artifacts of the specific training run. A surprising "aha moment" was that numerical quadrature was roughly two times more efficient than sampling-based inference and provided more robust results for training.

Conclusion

While the model handles heterogeneous data types well, balancing "latent channel capacity" against reconstruction accuracy requires careful hyperparameter tuning. Future work could involve extending this framework to handle temporal data (longitudinal patient records) or incorporating even more complex likelihood functions for specialized biological data. Additionally, while inducing points make the GP scalable, extremely large datasets might still pose a computational challenge compared to pure neural network approaches.

Summary updated

Abstract

Clinical patient records are an example of high-dimensional data that is typically collected from disparate sources and comprises of multiple likelihoods with noisy as well as missing values. In this work, we propose an unsupervised generative model that can learn a low-dimensional representation among the observations in a latent space, while making use of all available data in a heterogeneous data setting with missing values. We improve upon the existing Gaussian process latent variable model (GPLVM) by incorporating multiple likelihoods and deep neural network parameterised back-constraints to create a non-linear dimensionality reduction technique for heterogeneous data. In addition, we develop a variational inference method for our model that uses numerical quadrature. We establish the effectiveness of our model and compare against existing GPLVM methods on a standard benchmark dataset as well as on clinical data of Parkinson’s disease patients treated at the HUS Helsinki University Hospital.
BibTeX
@inproceedings{ramchandran2021latent,
  author = {Siddharth Ramchandran and Miika Koskinen and Harri Lähdesmäki},
  title = {{Latent Gaussian process with composite likelihoods and numerical quadrature}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
  url = {https://openreview.net/forum?id=78El4tp2SOi},
}
Figure