Time Series as Images: Vision Transformer for Irregularly Sampled Time Series

Zekun Li, Shiyang Li, Xifeng Yan

TL;DR

ViTST (Vision Time Series Transformer) treats multivariate, irregularly sampled time series as a computer vision problem by converting them into line graph images. By leveraging pre-trained Vision Transformers, this minimalist approach outperforms highly specialized state-of-the-art models and demonstrates extraordinary robustness, improving F1 scores by over 40% in scenarios where half the sensors are missing.

What Problem Does This Paper Address?

In fields like healthcare, time series data is rarely "clean." Sensors fail, patients are monitored at irregular intervals, and different variables (like heart rate vs. lab results) are recorded at different frequencies. This results in irregularly sampled time series—sparse, messy data that breaks traditional deep learning models like LSTMs or standard Transformers, which typically expect regular, fixed-size numerical grids.

Existing solutions are often "bespoke"—highly specialized architectures involving complex imputation, differential equations, or graph neural networks. These models require significant domain expertise to design and often struggle with extreme sparsity or "leave-sensors-out" scenarios, where certain variables available during training are missing during real-world testing. The authors of this paper ask a simple but provocative question: If humans can analyze complex data by looking at a chart, why can't a Vision Transformer (ViT) do the same?

Methods

The proposed framework, ViTST, follows a two-step "minimalist" pipeline:

  1. Time Series to Image Transformation: Each variable in a multivariate time series is plotted as an individual line graph. These graphs use markers for observed points and linear interpolation for gaps. All individual graphs are then arranged into a single grid-style RGB image (e.g., a 6x6 grid for 36 variables).
  2. Vision Transformer Backbone: This composite image is fed into a pre-trained Vision Transformer. The authors primarily use the Swin Transformer, which is particularly well-suited for this task. Its "shifted window" mechanism allows the model to capture both local patterns (the temporal dynamics within a single variable's graph) and global correlations (how different variables interact across the grid).
Figure 1: An illustration of our approach ViTST. The example is from a healthcare dataset P12 [12], which provides the irregularly sampled observations of 36 variables for patients (we only show 4 variables here for simplicity). Each column in the table is an observation of a variable, with the observed time and value. We plot separate line graphs for each variable and arrange them into a single image, which is then fed into the vision transformer for classification.

To handle non-temporal data (like a patient's age or gender), the authors encode these "static features" using a pre-trained RoBERTa model and concatenate the resulting text embeddings with the visual features before the final classification head.

Datasets

The authors evaluated ViTST on three primary irregularly sampled datasets:

  • P19 & P12: Healthcare datasets from the PhysioNet challenges, focused on predicting sepsis and mortality. These are characterized by high missingness (up to 94.9%) and class imbalance.
  • PAM: A human activity recognition dataset with 17 sensor variables.

Additionally, they tested the model's versatility on 10 regular multivariate time series datasets from the UEA archive, including challenges like EigenWorms (extremely long sequences) and PEMS-SF (nearly 1,000 variables).

Key Insights and Ideas

  • Knowledge Transfer from Natural Images: One of the most surprising findings is that a ViT pre-trained on natural images (like dogs and cars) can effectively "read" synthetic line graphs. The model transfers its general ability to recognize shapes, slopes, and spatial relationships to the domain of data visualization.
  • Visual Robustness: Numerical models often fail when a specific input feature disappears. However, because ViTST treats the entire variable set as a visual scene, it is remarkably resilient. Even if a variable's graph is mostly empty, the model can still "see" the context, much like a human doctor looking at a partially blank medical chart.
  • Shifted Windows for Cross-Variable Logic: The use of the Swin Transformer is a clever architectural choice. As shown in the figure below, the shifted window allows the attention mechanism to span across the boundaries of individual sub-graphs, enabling the model to learn inter-variable dependencies.
Figure 2: Illustration of the shifted window approach in Swin Transformer. Self-attention is calculated within each window (grey box). When the window is contained within a single line graph, it captures local interactions. After shifting, the window includes patches from different line graphs, allowing for the model- ing of global cross-variable interactions.

Key Results

  • SOTA Performance: ViTST outperformed specialized baselines (like Raindrop and mTAND) across healthcare and activity datasets. On the PAM dataset, it saw a 7.3% absolute increase in accuracy.
  • Extreme Robustness: In the "leave-sensors-out" setting—where 50% of the sensors are masked during testing—ViTST maintained an F1 score above 80%, while the best specialized baseline (Raindrop) plummeted to below 40%. This represents a staggering 42.8% improvement.
  • Versatility: Despite being designed for irregular data, ViTST achieved the second-best average accuracy on regular time series datasets, proving it can serve as a general-purpose framework.

Limitations and Future Work

  • Computational Overhead: Converting data to images and running a large ViT is computationally expensive. As shown in the paper's tables, ViTST's inference time is significantly higher than simpler numerical models (e.g., 44s vs 0.2s for the P19 test set).
  • Visualization Sensitivity: The performance is sensitive to how the graphs are plotted. The authors found that using distinctive colors for different variables was crucial, while adding tick labels or axes was actually superfluous and sometimes distracting for the model.
  • Future Directions: The authors suggest exploring more sophisticated "visual prompts" beyond simple line graphs and investigating self-supervised pre-training specifically on visualized data to further close the performance gap.

Summary updated

Abstract

Irregularly sampled time series are increasingly prevalent, particularly in medical domains. While various specialized methods have been developed to handle these irregularities, effectively modeling their complex dynamics and pronounced sparsity remains a challenge. This paper introduces a novel perspective by converting irregularly sampled time series into line graph images, then utilizing powerful pre-trained vision transformers for time series classification in the same way as image classification. This method not only largely simplifies specialized algorithm designs but also presents the potential to serve as a universal framework for time series modeling. Remarkably, despite its simplicity, our approach outperforms state-of-the-art specialized algorithms on several popular healthcare and human activity datasets. Especially in the rigorous leave-sensors-out setting where a portion of variables is omitted during testing, our method exhibits strong robustness against varying degrees of missing observations, achieving an impressive improvement of 42.8% in absolute F1 score points over leading specialized baselines even with half the variables masked. Code and data are available at https://github.com/Leezekun/ViTST
BibTeX
@inproceedings{li2023time,
  author = {Zekun Li and Shiyang Li and Xifeng Yan},
  title = {{Time Series as Images: Vision Transformer for Irregularly Sampled Time Series}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=ZmeAoWQqe0},
}
Figure