Abstract
Longitudinal datasets measured repeatedly over time from individual subjects, arise in many biomedical, psychological, social, and other studies. A common approach to analyse high-dimensional data that contains missing values is to learn a low-dimensional representation using variational autoencoders (VAEs). However, standard VAEs assume that the learnt representations are i.i.d., and fail to capture the correlations between the data samples. We propose the Longitudinal VAE (L-VAE), that uses a multi-output additive Gaussian process (GP) prior to extend the VAE’s capability to learn structured low-dimensional representations imposed by auxiliary covariate information, and derive a new KL divergence upper bound for such GPs. Our approach can simultaneously accommodate both time-varying shared and random effects, produce structured low-dimensional representations, disentangle effects of individual covariates or their interactions, and achieve highly accurate predictive performance. We compare our model against previous methods on synthetic as well as clinical datasets, and demonstrate the state-of-the-art performance in data imputation, reconstruction, and long-term prediction tasks.
Introduction
Variational Autoencoders (VAEs) are powerful tools for learning low-dimensional representations of high-dimensional data. However, a significant limitation of standard VAEs is the assumption that data samples are independent and identically distributed (i.i.d.). This assumption is frequently violated in longitudinal studies, such as clinical trials or psychological assessments, where measurements from the same individual are correlated over time. While previous works like GP-VAE have attempted to incorporate Gaussian Processes (GPs) to model these dependencies, they often fail to integrate auxiliary covariates (e.g., age, sex), struggle with scalability due to the $O(N^3)$ complexity of GPs, and cannot effectively disentangle shared population trends from individual variations.
Methods
The Longitudinal Variational Autoencoder (L-VAE) addresses these challenges by replacing the standard independent latent prior with a multi-output additive GP prior.
In the L-VAE framework, the latent space $Z$ is governed by a GP where the covariance function is a sum of multiple kernels. This additive structure allows the model to explicitly incorporate auxiliary covariates $X$ (like patient ID or age). For instance, specific kernels can model population-level effects (shared trends) while others capture instance-level effects (individual deviations). This design facilitates the learning of "disentangled" representations where different latent dimensions correspond to specific covariates.
To overcome the computational bottlenecks of GPs, the authors derived a novel KL divergence upper bound. By utilizing inducing points and exploiting the additive kernel structure, the model becomes scalable to large datasets through mini-batch Stochastic Gradient Descent (SGD), making it practical for real-world applications with tens of thousands of observations.
Experiments and Results
The L-VAE was evaluated across three diverse datasets: Rotated MNIST, a novel Health MNIST dataset, and the Physionet Challenge 2012 ICU dataset.
Health MNIST was specifically designed to simulate longitudinal clinical data, using digit types for sex, rotations for disease progression, and spatial shifts for aging. L-VAE demonstrated superior performance in both missing value imputation and long-term prediction compared to GP-VAE and RNN-based models like BRITS.
On the Physionet mortality prediction task, L-VAE achieved a higher AUROC than standard VAEs and other GP-based variants. Furthermore, latent space visualizations confirmed that the model successfully disentangled different factors, such as separating "disease progression" (rotation) from "aging" (spatial shift) in the Health MNIST experiments.
Conclusion
The L-VAE provides a scalable and interpretable framework for analyzing high-dimensional longitudinal data. By integrating additive GP priors, it effectively captures complex correlations and disentangles covariate effects. Future research may explore more sophisticated variational distributions and automated methods for kernel structure discovery to further enhance the model's flexibility and ease of use.
Summary updated