TL;DR
This paper proposes a novel approach, ViTST, that transforms irregularly sampled time series into line graph images and leverages pre-trained Vision Transformers (ViTs) for classification. This method simplifies modeling by treating time series as images, achieving state-of-the-art performance on healthcare and human activity datasets, and demonstrating remarkable robustness to missing data.
What Problem Does This Paper Address?
Irregularly sampled time series, where data points are not collected at fixed intervals, are increasingly common, especially in critical domains like healthcare. While specialized methods exist to handle these irregularities, they often require significant domain expertise and complex model designs. Modeling the intricate dynamics and sparsity of such data remains a challenge. The authors aim to bridge this gap by proposing a simpler, more generalizable approach that can harness the power of modern computer vision models.
Methods
The core of ViTST lies in its two-step process:
- Time Series to Image Transformation: Each multivariate time series is converted into a single RGB image. This is achieved by plotting each variable as a line graph, where the x-axis represents time and the y-axis represents the observed value. These individual line graphs are then arranged into a grid to form a composite image. The authors found that simply plotting the observed points and connecting them with lines, without complex interpolation or normalization, was sufficient.
- Vision Transformer for Classification: A pre-trained Vision Transformer (specifically, Swin Transformer was used as the default) is then used to classify these generated images, treating them as standard image classification tasks. This leverages the powerful feature extraction capabilities of ViTs, which have excelled in tasks like image recognition.
Datasets
The authors evaluated ViTST on three widely used datasets:
- P19 (PhysioNet Sepsis Early Prediction Challenge 2019): Healthcare dataset for sepsis prediction, with 34 sensor variables and 6 demographic features. Highly imbalanced with a 94.9% missing ratio.
- P12 (PhysioNet Mortality Prediction Challenge 2012): Healthcare dataset for patient mortality prediction, with 36 sensor variables and 6 static demographic features. Highly imbalanced with an 88.4% missing ratio.
- PAM (PAMAP2 Physical Activity Monitoring): Human activity dataset with 17 sensor variables. Approximately balanced classes, with a 60.0% missing ratio.
They also conducted experiments on ten representative regular multivariate time series datasets from the UEA Time Series Classification Archive to demonstrate the approach's versatility.
Key Insights and Ideas
- Time Series as Images is a Powerful Abstraction: The most striking insight is that by simply visualizing time series data as line graphs and feeding these images to a ViT, one can achieve state-of-the-art results without needing specialized time series architectures. This suggests that the visual patterns in time series are rich enough for powerful image models to learn from.
- Leveraging Pre-trained Vision Models: The success of ViTST highlights the transferability of knowledge from models pre-trained on natural images to synthetic time series images. This opens doors to applying cutting-edge computer vision techniques to time series analysis.
- Robustness to Missing Data: ViTST demonstrates exceptional resilience to missing observations, significantly outperforming specialized methods in "leave-sensors-out" scenarios where a substantial portion of variables is masked. This is crucial for real-world applications where data is often incomplete.
- Simplicity and Generalizability: The approach is remarkably simple, requiring minimal domain-specific engineering for time series modeling. This suggests ViTST could serve as a universal framework for various time series tasks.
Key Results
- State-of-the-Art Performance: ViTST consistently outperformed specialized methods on the P19, P12, and PAM datasets, achieving improvements in AUROC, AUPRC, Accuracy, Precision, Recall, and F1 scores. For instance, on the PAM dataset, it achieved a 7.3% improvement in accuracy.
- Exceptional Robustness: In leave-sensors-out settings, ViTST achieved an impressive 42.8% absolute improvement in F1 score over the best baseline when half the variables were masked.
- Comparable to Regular Time Series Models: ViTST also performed competitively on regular time series datasets, demonstrating its broad applicability.
- ViTs Outperform CNNs: When comparing different backbone models, pre-trained ViTs (Swin and ViT) significantly outperformed a pre-trained CNN (ResNet) for this task, suggesting ViTs' superior ability to capture spatial information.
Limitations and Future Work
- Image Creation Nuances: While the default image creation strategy worked well, the authors explored variations like removing interpolation or markers, which sometimes led to minor performance drops, indicating that certain visual cues are important.
- Computational Cost: ViTST's inference time is higher than some non-vision baselines, though considered acceptable for ML practice.
- Understanding ViT's Mechanism: The exact reasons why ViTs are so effective at capturing temporal patterns in visualized time series require further exploration.
- Alternative Visualizations: The authors suggest exploring other visualization techniques beyond line graphs and investigating more advanced image representations.
- Multi-modal Integration: The framework is well-suited for integrating other modalities (like text from EHRs), and future work could explore more sophisticated multi-modal fusion techniques.
Summary updated