Bidirectional Generative Pre-training for Improving Healthcare Time-series Representation Learning

Yue Li, Ziyang Song, Qincheng Lu, He Zhu, David L. Buckeridge, Yue Li 0017Mlhc 2024

TL;DR

This paper introduces BiTimelyGPT, a novel pre-training method for healthcare time-series data that addresses limitations of existing approaches by incorporating bidirectional context and preserving data distribution. BiTimelyGPT achieves state-of-the-art performance on various downstream tasks, demonstrating its effectiveness in learning robust representations from complex medical data.

What Problem Does This Paper Address?

Learning meaningful representations from healthcare time-series data, such as biosignals and electronic health records (EHRs), is crucial for tasks like disease diagnosis and predicting patient outcomes. However, current pre-training methods for time-series data have limitations:

  • Unidirectional Pre-training: Models like GPT, while effective for generative tasks, use causal attention, limiting their ability to capture bidirectional dependencies essential for discriminative tasks.
  • Masking-based Methods: Techniques like PatchTST mask segments by setting them to zero. This creates a "distribution shift," introducing noise and degrading representation quality, especially problematic for time-series where signal integrity is key.
  • Dropping-based Methods: Approaches like CRT discard segments to avoid distribution shift but risk disrupting important discriminative subsequences (shapelets) within the time-series.
  • Low-Rank Attention: Standard self-attention mechanisms can lead to low-rank attention matrices, limiting their expressiveness and ability to capture complex relationships.

The authors aim to overcome these challenges by developing a pre-training strategy that preserves data distribution, captures bidirectional contexts, and utilizes expressive, full-rank attention matrices.

Methods

The core innovation is the Bidirectional Timely Generative Pre-trained Transformer (BiTimelyGPT). Its key components include:

  • Next-Previous-Token Prediction: This novel pre-training task is designed to preserve the original data distribution and time-series shapelets. It involves predicting both the next token (forward) and the previous token (backward) in alternating layers.
  • Bidirectional Alternating AutoRegressive Modeling (BAAR) Framework: This framework enables deep bidirectional representation learning by alternating information flow between left-to-right (forward) and right-to-left (backward) across layers. Odd-indexed layers handle forward prediction, while even-indexed layers handle backward prediction.
  • Bidirectional Retention Layer: BiTimelyGPT utilizes a modified retention mechanism that alternates between forward and backward attention across layers. This allows for deep bidirectional context learning.
  • Full-Rank Attention Matrices: By alternating forward and backward retention, BiTimelyGPT achieves full-rank attention matrices. This is crucial for enhanced expressiveness and overcoming the limitations of low-rank attention found in other models.

The architecture involves a convolution-subsampling tokenizer, input projection, L bidirectional retention layers, and an output projection for the pre-training task. For downstream tasks, the [SOS] token from the final layer is used as a sequence representation.

Datasets

The paper utilizes several publicly available datasets for pre-training and downstream tasks:

Pre-training Datasets: * Sleep-EDF: Large-scale biosignal dataset with 7 types of biosignals. * PTB-XL: Large-scale electrocardiogram (ECG) dataset with 12 types of signals. * PPG-Dalia: Dataset for photoplethysmograph (PPG) data, used for heart rate estimation.

Downstream Task Datasets: * Biosignals: Epilepsy, EMG, RR, HR, SpO2. * Healthcare Administrative Data: PopHR (Population Health Record) dataset, containing irregularly sampled time series of clinical records (ICD codes converted to PheCodes).

Key Insights and Ideas

  • Preserving Data Integrity is Key: Unlike masking-based methods that introduce artificial zeros, BiTimelyGPT's Next-Previous-Token Prediction task preserves the original signal distribution and shapelets, leading to more robust representations.
  • Bidirectionality is Crucial for Healthcare: Healthcare time-series often contain complex temporal dependencies that require understanding context from both past and future. BiTimelyGPT's BAAR framework effectively captures these bidirectional relationships.
  • Full-Rank Attention Matters for Expressiveness: The paper demonstrates that achieving full-rank attention matrices, through the alternating forward/backward retention mechanism, leads to more expressive representations compared to low-rank alternatives.
  • [SOS] Token as a Powerful Sequence Representation: The study highlights that the [SOS] token from the final layer, aggregated by backward attention, serves as an effective sequence representation for downstream tasks.

Key Results

  • Superior Performance: BiTimelyGPT achieved state-of-the-art performance on several downstream tasks, including classification on Sleep-EDF and Epilepsy datasets, and regression on IEEEPPG, HR, and SpO2 datasets.
  • Improved Accuracy: On the Sleep-EDF classification task, BiTimelyGPT achieved 90.4% accuracy, outperforming unidirectional models like TimelyGPT (89.2%) and masking-based PatchTST (89.6%).
  • Effective Bidirectional Context: Qualitative analysis (attention heatmaps) showed BiTimelyGPT focusing attention more effectively on discriminative regions compared to unidirectional models.
  • Expressive Attention: Singular value decomposition analysis demonstrated that BiTimelyGPT's attention matrices are more uniformly distributed, indicating higher rank and greater expressiveness than baselines like CRT and TimelyBERT.
  • Generalizability: BiTimelyGPT also showed strong performance on the irregularly sampled PopHR dataset for chronic disease classification, achieving an average AUPRC of 60.5%, surpassing specialized methods.
  • Ablation Study: The Next-Previous-Token Prediction strategy, combined with the BAAR framework, was shown to be critical for BiTimelyGPT's performance, outperforming simpler next-token prediction or masking-based pre-training.

Limitations and Future Work

  • In-Domain Focus: The current study concentrates on in-domain healthcare time-series data with similar biological patterns. The capacity to transfer learned representations to out-of-distribution biosignals and other data modalities remains untested.
  • Broader Applicability of BAAR: While the BAAR framework has shown effectiveness with the Retention mechanism, its potential benefits for other autoregressive models need further exploration.
  • Future Directions: The authors suggest exploring the transferability of BiTimelyGPT to out-of-distribution healthcare data and investigating its application to a wider range of autoregressive models.

Summary updated

BibTeX
@inproceedings{li2024bidirectional,
  author = {Yue Li and Ziyang Song and Qincheng Lu and He Zhu and David L. Buckeridge and Yue Li},
  title = {{Bidirectional Generative Pre-training for Improving Healthcare Time-series Representation Learning}},
  booktitle = {Mlhc},
  year = {2024},
}
Figure