Self-supervised Transformer for Multivariate Clinical Time-Series with Missing Values

Sindhu Tipirneni, Chandan K. Reddy

Introduction: The Challenge of Clinical Data

In clinical settings, particularly Intensive Care Units (ICUs), data is notoriously difficult for standard machine learning models to process. Unlike clean, regularly sampled datasets, ICU data is characterized by irregular intervals and extreme sparsity. For instance, one patient might have their heart rate checked every 15 minutes, while another has it checked every 4 hours. Furthermore, not every lab test is ordered for every patient at every time step, leading to many "missing" values in a traditional data matrix.

Example of irregular and sparse multivariate clinical time-series.

Traditional approaches often rely on "binning" data into fixed time intervals or "imputation" to fill in missing values. However, these methods can suppress fine-grained details or introduce artificial noise that degrades model performance.

The STraTS Architecture

The Self-supervised Transformer for Time-Series (STraTS) architecture addresses these issues by moving away from the traditional data matrix approach. Instead of a fixed grid, STraTS treats time-series data as a set of observation triplets $(t_i, f_i, v_i)$, where $t_i$ is the time of observation, $f_i$ is the clinical variable, and $v_i$ is the measured value. This allows the model to process only the data that actually exists, ignoring the "gaps" entirely.

The STraTS model architecture featuring triplet embeddings and self-attention.

Key Methodological Innovations

  • Continuous Value Embedding (CVE): Rather than discretizing time or values into buckets, STraTS uses a novel "one-to-many" feed-forward network to learn continuous mappings. This preserves the precision of raw measurements, such as the exact timing of a heart rate reading.
  • Self-Supervised Pre-training: To mitigate the lack of labeled data, the model is first trained on a forecasting task. It learns to predict future measurements based on an initial observation window. This "proxy task" allows the model to learn complex patient dynamics from vast amounts of unlabeled data before it ever sees a mortality label.
  • Transformer-Based Fusion: The model uses a Transformer's multi-head self-attention mechanism to naturally weight relevant past observations, regardless of their timing, to form a comprehensive patient representation.

Experimental Evaluation

STraTS was evaluated on two major ICU benchmarks: MIMIC-III (over 40,000 stays) and the PhysioNet Challenge 2012 (nearly 12,000 stays). The primary task was mortality prediction based on the first 24 to 48 hours of admission.

Key Results

  • State-of-the-Art Performance: STraTS consistently outperformed baselines like GRU-D and InterpNet. On MIMIC-III, it achieved a PR-AUC of 0.577, a significant jump over the best baseline (GRU-D at 0.544).
  • Data Efficiency: When trained on only 10% of available labels, STraTS maintained a much higher performance lead over other models. This confirms that self-supervision effectively "multiplies" the value of limited labeled data.
  • Interpretability: A variant called I-STraTS provides "contribution scores," allowing clinicians to see exactly which measurements (like elevated Lactate or age) drove a high-risk prediction, matching clinical knowledge.

Conclusion and Future Directions

STraTS demonstrates that clinical time-series can be modeled effectively without lossy aggregation or noisy imputation. By leveraging the flexibility of Transformers and the power of self-supervision, the model achieves high accuracy even with sparse, irregularly sampled data. While the current architecture faces computational challenges as the number of observations grows, future work focusing on sparse attention mechanisms could scale these benefits to even longer clinical histories.

Summary updated

BibTeX
@article{tipirneni2022self,
  author = {Sindhu Tipirneni and Chandan K. Reddy},
  title = {{Self-supervised Transformer for Multivariate Clinical Time-Series with Missing Values}},
  journal = {ACM Trans. Knowl. Discov. Data},
  year = {2022},
  doi = {10.1145/3516367},
}
Figure