Time Series as Images: Vision Transformer for Irregularly Sampled Time Series

Zekun Li, Shiyang Li, Xifeng Yan

TL;DR

This paper proposes a novel approach, ViTST, that transforms irregularly sampled time series into line graph images and leverages pre-trained Vision Transformers (ViTs) for classification. This method simplifies modeling by treating time series as images, achieving state-of-the-art performance on healthcare and human activity datasets, and demonstrating remarkable robustness to missing data.

What Problem Does This Paper Address?

Irregularly sampled time series, where data points are not collected at fixed intervals, are increasingly common, especially in critical domains like healthcare. While specialized methods exist to handle these irregularities, they often require significant domain expertise and complex model designs. Modeling the intricate dynamics and sparsity of such data remains a challenge. The authors aim to bridge this gap by proposing a simpler, more generalizable approach that can harness the power of modern computer vision models.

Methods

The core of ViTST lies in its two-step process:

  1. Time Series to Image Transformation: Each multivariate time series is converted into a single RGB image. This is achieved by plotting each variable as a line graph, where the x-axis represents time and the y-axis represents the observed value. These individual line graphs are then arranged into a grid to form a composite image. The authors found that simply plotting the observed points and connecting them with lines, without complex interpolation or normalization, was sufficient.
  2. Vision Transformer for Classification: A pre-trained Vision Transformer (specifically, Swin Transformer was used as the default) is then used to classify these generated images, treating them as standard image classification tasks. This leverages the powerful feature extraction capabilities of ViTs, which have excelled in tasks like image recognition.

Datasets

The authors evaluated ViTST on three widely used datasets:

  • P19 (PhysioNet Sepsis Early Prediction Challenge 2019): Healthcare dataset for sepsis prediction, with 34 sensor variables and 6 demographic features. Highly imbalanced with a 94.9% missing ratio.
  • P12 (PhysioNet Mortality Prediction Challenge 2012): Healthcare dataset for patient mortality prediction, with 36 sensor variables and 6 static demographic features. Highly imbalanced with an 88.4% missing ratio.
  • PAM (PAMAP2 Physical Activity Monitoring): Human activity dataset with 17 sensor variables. Approximately balanced classes, with a 60.0% missing ratio.

They also conducted experiments on ten representative regular multivariate time series datasets from the UEA Time Series Classification Archive to demonstrate the approach's versatility.

Key Insights and Ideas

  • Time Series as Images is a Powerful Abstraction: The most striking insight is that by simply visualizing time series data as line graphs and feeding these images to a ViT, one can achieve state-of-the-art results without needing specialized time series architectures. This suggests that the visual patterns in time series are rich enough for powerful image models to learn from.
  • Leveraging Pre-trained Vision Models: The success of ViTST highlights the transferability of knowledge from models pre-trained on natural images to synthetic time series images. This opens doors to applying cutting-edge computer vision techniques to time series analysis.
  • Robustness to Missing Data: ViTST demonstrates exceptional resilience to missing observations, significantly outperforming specialized methods in "leave-sensors-out" scenarios where a substantial portion of variables is masked. This is crucial for real-world applications where data is often incomplete.
  • Simplicity and Generalizability: The approach is remarkably simple, requiring minimal domain-specific engineering for time series modeling. This suggests ViTST could serve as a universal framework for various time series tasks.

Key Results

  • State-of-the-Art Performance: ViTST consistently outperformed specialized methods on the P19, P12, and PAM datasets, achieving improvements in AUROC, AUPRC, Accuracy, Precision, Recall, and F1 scores. For instance, on the PAM dataset, it achieved a 7.3% improvement in accuracy.
  • Exceptional Robustness: In leave-sensors-out settings, ViTST achieved an impressive 42.8% absolute improvement in F1 score over the best baseline when half the variables were masked.
  • Comparable to Regular Time Series Models: ViTST also performed competitively on regular time series datasets, demonstrating its broad applicability.
  • ViTs Outperform CNNs: When comparing different backbone models, pre-trained ViTs (Swin and ViT) significantly outperformed a pre-trained CNN (ResNet) for this task, suggesting ViTs' superior ability to capture spatial information.

Limitations and Future Work

  • Image Creation Nuances: While the default image creation strategy worked well, the authors explored variations like removing interpolation or markers, which sometimes led to minor performance drops, indicating that certain visual cues are important.
  • Computational Cost: ViTST's inference time is higher than some non-vision baselines, though considered acceptable for ML practice.
  • Understanding ViT's Mechanism: The exact reasons why ViTs are so effective at capturing temporal patterns in visualized time series require further exploration.
  • Alternative Visualizations: The authors suggest exploring other visualization techniques beyond line graphs and investigating more advanced image representations.
  • Multi-modal Integration: The framework is well-suited for integrating other modalities (like text from EHRs), and future work could explore more sophisticated multi-modal fusion techniques.

Summary updated

Abstract

Irregularly sampled time series are becoming increasingly prevalent in various domains, especially in medical applications. Although different highly-customized methods have been proposed to tackle irregularity, how to effectively model their complicated dynamics and high sparsity is still an open problem. This paper studies the problem from a whole new perspective: transforming irregularly sampled time series into line graph images and adapting powerful vision transformers to perform time series classification in the same way as image classification. Our approach largely simplifies algorithm designs without assuming prior knowledge and can be potentially extended as a general-purpose framework. Despite its simplicity, we show that it substantially outperforms state-of-the-art specialized algorithms on several popular healthcare and human activity datasets. Especially in the challenging leave-sensors-out setting where a subset of variables is masked during testing, the performance improvement is up to 54.0\% in absolute F1 score points. Our code and data are available at \url{https://github.com/Leezekun/ViTST}.
BibTeX
@inproceedings{li2023time,
  author = {Zekun Li and Shiyang Li and Xifeng Yan},
  title = {{Time Series as Images: Vision Transformer for Irregularly Sampled Time Series}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=ZmeAoWQqe0},
}
Figure