Time Series Representations for Classification Lie Hidden in Pretrained Vision Transformers

Simon Roschmann, Quentin Bouniot, Vasilii Feofanov, Ievgen Redko, Zeynep Akata

TL;DR

The authors introduce TiViT (Time Vision Transformer), a framework that reformats 1D time series data into 2D images to leverage the rich, generalizable representations of frozen, pretrained Vision Transformers (ViTs). By extracting features from intermediate layers where the intrinsic dimensionality is highest, TiViT outperforms state-of-the-art time series foundation models on standard classification benchmarks without requiring any fine-tuning.

What Problem Does This Paper Address?

While foundation models have revolutionized NLP and computer vision, progress in the time series domain has been bottlenecked by data scarcity. Training large-scale Time Series Foundation Models (TSFMs) from scratch is challenging because massive, high-quality, publicly available time series datasets are hard to come by.

To circumvent this, researchers have tried repurposing pretrained language or vision models. However, previous attempts often required computationally expensive fine-tuning, were restricted to narrow tasks (like univariate forecasting), or underperformed compared to specialized TSFMs. This paper addresses a fundamental question: Can we effectively reuse powerful, frozen vision representations for time series classification, and if so, why does it work?

Methods

The TiViT framework consists of a simple yet highly effective pipeline:

  1. Time Series-to-Image Transformation: A 1D time series is normalized using robust scaling and segmented into patches of length $P = \sqrt{T}$ (where $T$ is the series length). These patches are stacked vertically to form a 2D matrix, which is then replicated across three channels to create a grayscale image and resized to match the standard input resolution of a ViT.
  2. Feature Extraction via Frozen ViTs: The generated image is fed into a frozen, pretrained ViT backbone (such as OpenCLIP, DINOv2, or SigLIP 2).
  3. Intermediate Layer Aggregation: Instead of using the final layer's output or the standard [CLS] token, TiViT extracts embeddings from an intermediate layer (typically between 40% and 70% of the model's total depth) and averages the representations across all tokens.
  4. Classification: A simple linear classifier (logistic regression) is trained on top of these static embeddings.
  5. TSFM Fusion: To maximize performance, TiViT representations can be directly concatenated with embeddings from specialized time series models before classification.
The TiViT pipeline transforms 1D signals into 2D images to extract hidden features from frozen ViT layers.

Datasets

The framework was rigorously evaluated on two major archives:

  • UCR Archive: 128 univariate time series datasets spanning various domains (e.g., healthcare, robotics, sensor data).
  • UEA Archive: 27 multivariate time series datasets (after filtering out 3 datasets due to short sequence lengths or small test sizes).

For the baseline comparison models, a pretraining corpus of 100,000 samples was curated from diverse public datasets including ECG, EMG, Epilepsy, FD-A/B, Gesture, HAR, and SleepEEG.

Key Insights and Ideas

  • The Theoretical Edge of 2D Patching: The authors mathematically prove that reshaping a time series into a 2D grid and applying 2D patching reduces sample complexity during training compared to traditional 1D patching. 2D patching spreads discriminative, label-relevant signals across a larger fraction of tokens, making it significantly easier for the Transformer's attention mechanism to lock onto important patterns.
An illustration of how 2D patching increases the number of label-relevant tokens compared to 1D patching.
  • The "Sweet Spot" is in the Middle: The final layers of vision models are heavily optimized for high-level visual semantics (e.g., distinguishing a cat from a dog), which destroys the abstract structural properties of a reshaped time series. The authors discovered that intermediate layers retain the highest intrinsic dimensionality—meaning they preserve the most complex, uncompressed structural information, making them ideal for non-visual downstream tasks.
Validation accuracy and intrinsic dimension across relative layer depths for different ViT backbones.
  • Strong Cross-Domain Complementarity: Representation space alignment analysis revealed that frozen ViTs and specialized TSFMs (like Mantis) extract fundamentally different features from the same data. Because their representation spaces have very low alignment, they are highly complementary.

Key Results

  • Outperforming Specialized TSFMs: On the 128 UCR datasets, TiViT (using an OpenCLIP ViT-H/14 backbone) achieved a mean linear classification accuracy of 81.3%, outperforming the state-of-the-art in-domain TSFM, Mantis (80.1%).
  • Better Together (Feature Fusion): Concatenating TiViT features with Mantis features yielded a substantial performance boost, reaching 83.0% accuracy on UCR and 73.7% on UEA.
Comparison of classification accuracy across 155 datasets from the UCR and UEA archives.
  • Zero-Shot Capability: Evaluated with a simple nearest-centroid classifier (requiring no logistic regression training), TiViT achieved a zero-shot accuracy of 71.6% on the UCR benchmark, demonstrating excellent feature generalizability.

Limitations and Future Work

While TiViT delivers impressive results, the authors note a few limitations:

  1. Channel Independence: For multivariate datasets, TiViT processes each channel independently and simply concatenates the final embeddings. It does not explicitly model cross-channel correlations.
  2. Simplistic Fusion: The integration of TSFM and ViT features relies on basic concatenation; more sophisticated multimodal fusion strategies remain unexplored.
  3. Frozen Backbone: The vision weights are kept completely frozen. Unfreezing or parameter-efficient fine-tuning (PEFT) of the ViT backbone could potentially unlock even higher performance.

Summary updated

Abstract

Time series classification is a fundamental task in healthcare and industry, yet the development of time series foundation models (TSFMs) remains limited by the scarcity of publicly available time series datasets. In this work, we propose Time Vision Transformer (TiViT), a framework that converts time series into images to leverage the representational power of frozen Vision Transformers (ViTs) pretrained on large-scale image datasets. First, we theoretically motivate our approach by analyzing the 2D patching of ViTs for time series, showing that it can increase the number of label-relevant tokens and reduce the sample complexity. Second, we empirically demonstrate that TiViT achieves state-of-the-art performance on standard time series classification benchmarks by utilizing the hidden representations of large OpenCLIP models. We explore the structure of TiViT representations and find that intermediate layers with high intrinsic dimension are the most effective for time series classification. Finally, we assess the alignment between TiViT and TSFM representation spaces and identify a strong complementarity, with further performance gains achieved by combining their features. Our findings reveal a new direction for reusing vision representations in a non-visual domain. Code is available at https://github.com/ExplainableML/TiViT.
BibTeX
@misc{roschmann2025time,
  author = {Simon Roschmann and Quentin Bouniot and Vasilii Feofanov and Ievgen Redko and Zeynep Akata},
  title = {{Time Series Representations for Classification Lie Hidden in Pretrained Vision Transformers}},
  year = {2025},
  eprint = {2506.08641},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.08641},
}
Figure