SeqRisk: Transformer-augmented latent variable model for improved survival prediction with longitudinal data

Mine Ögretir, Miika Koskinen, Juha Sinisalo, Risto Renkonen, Harri Lähdesmäki

Abstract

In healthcare, risk assessment of different patient outcomes has for long time been based on survival analysis, i.e. modeling time-to-event associations. However, conventional approaches rely on data from a single time-point, making them suboptimal for fully leveraging longitudinal patient history and capturing temporal regularities. Focusing on clinical real-world data and acknowledging its challenges, we utilize latent variable models to effectively handle irregular, noisy, and sparsely observed longitudinal data. We propose SeqRisk, a method that combines variational autoencoder (VAE) or longitudinal VAE (LVAE) with a transformer encoder and Cox proportional hazards module for risk prediction. SeqRisk captures long-range interactions, improves patient trajectory representations, enhances predictive accuracy and generalizability, as well as provides partial explainability for sample population characteristics in attempts to identify high-risk patients. We demonstrate that SeqRisk performs competitively compared to existing approaches on both simulated and real-world datasets.

Introduction

Survival analysis—predicting the time until an event like a heart attack or hospital discharge—is a cornerstone of healthcare. However, real-world Electronic Health Records (EHRs) are notoriously "messy." They are longitudinal (collected over time), irregularly sampled, and often riddled with missing values. Traditional survival models, like the Cox proportional hazards model, often rely on a single "snapshot" of a patient’s health, failing to capture the rich temporal dynamics of a patient's history. SeqRisk addresses these challenges by integrating the generative power of Variational Autoencoders (VAEs) with the sequence-modeling prowess of Transformers.

Methodology: The SeqRisk Framework

SeqRisk is an end-to-end deep learning framework designed to handle raw, irregular longitudinal data through a "generative-discriminative" hybrid architecture.

The SeqRisk architecture integrates a VAE/LVAE encoder for latent representation with a Transformer-based survival head and a Cox regression module.

The architecture consists of three primary components:

  1. The Latent Module (VAE/LVAE): For standard data, it uses a VAE. For irregular longitudinal data, it employs a Longitudinal VAE (LVAE) with an additive multi-output Gaussian Process (GP) prior. This allows the model to explicitly account for temporal correlations and patient-specific variations, effectively "denoising" the input into a structured latent space.
  2. The Survival Head (Transformer): Instead of using a simple Recurrent Neural Network (RNN), SeqRisk uses an encoder-style Transformer to capture long-range interactions across the patient's history. An attention-pooling layer then collapses the sequence into a single vector representing the subject's risk.
  3. The Prediction Head: A non-linear extension of the Cox proportional hazards model yields the final risk score.

The model is trained end-to-end using a joint objective function that balances the Evidence Lower Bound (ELBO)—ensuring accurate data reconstruction—with the Cox partial likelihood, which ensures the latent space is discriminative for survival risk.

Experimental Evaluation

The authors evaluated SeqRisk on three distinct datasets: Survival MNIST (synthetic), PhysioNet Challenge 2012 (ICU data with >84% missingness), and a real-world Coronary Heart Disease (CHD) dataset from HUS Helsinki University Hospital characterized by extreme missingness (>98%).

Robustness to Data Sparsity

A key finding is SeqRisk's performance under extreme data sparsity. In real-world clinical settings, missingness can be a significant barrier to accurate prediction.

Comparison of Test C-Index across different levels of missingness. SeqRisk variants maintain higher and more stable performance compared to baselines as sparsity increases.

As shown in Figure 2, while traditional models and other deep learning baselines like Dynamic DeepHit show sharp performance declines as missingness increases, SeqRisk variants (particularly LVAE+Transformer) demonstrate remarkable stability. On the PhysioNet dataset, SeqRisk (VAE+Transformer) achieved a C-index of 0.789, outperforming Dynamic DeepHit (0.779) and RDSM (0.749). Additionally, SeqRisk (LVAE+Transformer) achieved an Integrated Brier Score (IBS) of 0.135, indicating better-calibrated probability estimates than most baselines.

Discussion and Conclusion

SeqRisk demonstrates that bridging generative latent variables with Transformers creates a robust framework for survival analysis. By learning continuous latent trajectories, the model handles irregular sampling and high missingness more effectively than previous approaches. While the current version focuses on single-event outcomes and lacks fine-grained feature-level interpretability, it represents a significant step toward leveraging the full complexity of longitudinal EHR data for improved patient risk stratification.

Summary updated

Abstract

In healthcare, risk assessment of different patient outcomes has for long time been based on survival analysis, i.e.\ modeling time-to-event associations. However, conventional approaches rely on data from a single time-point, making them suboptimal for fully leveraging longitudinal patient history and capturing temporal regularities. Focusing on clinical real-world data and acknowledging its challenges, we utilize latent variable models to effectively handle irregular, noisy, and sparsely observed longitudinal data. We propose SeqRisk, a method that combines variational autoencoder (VAE) or longitudinal VAE (LVAE) with a transformer encoder and Cox proportional hazards module for risk prediction. SeqRisk captures long-range interactions, improves patient trajectory representations, enhances predictive accuracy and generalizability, as well as provides partial explainability for sample population characteristics in attempts to identify high-risk patients. We demonstrate that SeqRisk performs competitively compared to existing approaches on both simulated and real-world datasets.
BibTeX
@misc{ogretir2024seqrisk,
  author = {Mine Ögretir and Miika Koskinen and Juha Sinisalo and Risto Renkonen and Harri Lähdesmäki},
  title = {{SeqRisk: Transformer-augmented latent variable model for improved survival prediction with longitudinal data}},
  year = {2024},
  eprint = {2409.12709},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.12709},
}
Figure