TDSTF: Transformer-based Diffusion probabilistic model for Sparse Time series Forecasting

Ping Chang, Huayu Li, Stuart F. Quan, Janet Roveda, Ao LiarXiv 2023 · arXiv · OpenReview

Abstract

In the intensive care unit (ICU), vital sign monitoring is critical, and an accurate predictive system is required. This study creates a novel model to forecast Heart Rate (HR), Systolic Blood Pressure (SBP), and Diastolic Blood Pressure (DBP) in the ICU. These vital signs are crucial for prompt interventions. We extracted 24,886 ICU stays from the MIMIC-III database to train and test the model. The proposed model, Transformer-based Diffusion Probabilistic Model for Sparse Time Series Forecasting (TDSTF), uses a deep learning technique called the Transformer. TDSTF showed state-of-the-art performance in predicting vital signs, outperforming other models in predicting distributions and being more computationally efficient. TDSTF achieved a Normalized Average Continuous Ranked Probability Score (NACRPS) of 0.4438 and a Mean Squared Error (MSE) of 0.4168, an improvement of 18.9% and 34.3% over the best baseline model, respectively.

Introduction: The Challenge of Sparse ICU Data

The Intensive Care Unit (ICU) is a data-rich environment, yet the Electronic Health Record (EHR) data is often sporadic and intermittent. In the MIMIC-III dataset, approximately 99.7% of potential data points are missing. Existing forecasting models like LSTMs or RNNs struggle with this level of sparsity, often requiring imputation or aggregation that introduces noise or destroys fine-grained temporal details. Furthermore, most models ignore the complex web of interrelated events—such as medication doses or lab results—that drive physiological changes.

Methods

TDSTF addresses these challenges by combining the generative power of Diffusion Probabilistic Models with the attention-driven architecture of Transformers. The approach rests on three pillars:

1. Diffusion Probabilistic Models

Instead of predicting a single value, the model learns to "denoise" Gaussian noise into a realistic distribution of future vital signs. This allows for probabilistic forecasting, providing a range of possible outcomes rather than just one "best guess."

The forward and reverse trajectories of the diffusion process for time series data.

2. The Triplet Representation

To solve the sparsity problem, the model abandons traditional matrix formats. Instead, it represents each data point as a "triplet" consisting of (feature, time, value) plus a mask bit. This allows the model to process only existing data, ignoring the "empty space" of missing records.

Conversion of sparse matrix data into a triplet format (feature, time, value, mask) to handle missing values efficiently.

3. Transformer Backbone

While previous diffusion models for time series used 1D-convolutional backbones, TDSTF utilizes a Transformer-based architecture. This uses self-attention to find correlations between different triplets regardless of their temporal distance, allowing the model to learn how specific medications interact with vital signs over time.

Key Results and Insights

TDSTF was evaluated using 24,886 unique ICU stays from the MIMIC-III database, using the first 30 minutes of data to forecast the following 10 minutes.

  • Accuracy: It achieved an 18.9% improvement in NACRPS and a 34.3% improvement in MSE over the best baseline model (CSDI).
  • Efficiency: The inference speed is more than 17 times faster than CSDI. While CSDI took nearly 15 seconds to generate a sample, TDSTF completed the task in less than 0.9 seconds.
  • Sparsity as a Strength: By using the triplet format, the model turns missing data into a computational advantage, processing only valid events.
  • Robustness: Qualitative results showed the model could successfully predict sudden changes, such as heart rate spikes, even with extremely thin input data.

Conclusion

TDSTF is an effective and efficient solution for forecasting vital signs in the ICU. While the current model is limited to 60 conditional triplets due to the quadratic memory cost of Transformers, it represents a significant step forward. Future work may involve scaling the model to handle longer patient histories and expanding the scope to include other vitals like oxygen saturation and breathing rate to provide a more holistic warning system for clinicians.

Summary updated

Abstract

Background and objective: In the intensive care unit (ICU), vital sign monitoring is critical, and an accurate predictive system is required. This study will create a novel model to forecast Heart Rate (HR), Systolic Blood Pressure (SBP), and Diastolic Blood Pressure (DBP) in ICU. These vital signs are crucial for prompt interventions for patients. We extracted $24,886$ ICU stays from the MIMIC-III database, which contains data from over $46$ thousand patients, to train and test the model. Methods: The model proposed in this study, areansformerin intensive careabilistic Model for Sparse Time Series Forecasting (TDSTF), uses a deep learning technique called the Transformer. The TDSTF model showed state-of-the-art performance in predicting vital signs in the ICU, outperforming other models' ability to predict distributions of vital signs and being more computationally efficient. The code is available at https://github.com/PingChang818/TDSTF. Results: The results of the study showed that TDSTF achieved a Normalized Average Continuous Ranked Probability Score (NACRPS) of $0.4438$ and a Mean Squared Error (MSE) of $0.4168$, an improvement of $18.9\%$ and $34.3\%$ over the best baseline model, respectively. Conclusion: In conclusion, TDSTF is an effective and efficient solution for forecasting vital signs in the ICU, and it shows a significant improvement compared to other models in the field.
BibTeX
@misc{chang2023tdstf,
  author = {Ping Chang and Huayu Li and Stuart F. Quan and Janet Roveda and Ao Li},
  title = {{TDSTF: Transformer-based Diffusion probabilistic model for Sparse Time series Forecasting}},
  year = {2023},
  eprint = {2301.06625},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.06625},
}
Figure