TL;DR
The authors introduce TiViT (Time Vision Transformer), a framework that reformats 1D time series data into 2D images to leverage the rich, generalizable representations of frozen, pretrained Vision Transformers (ViTs). By extracting features from intermediate layers where the intrinsic dimensionality is highest, TiViT outperforms state-of-the-art time series foundation models on standard classification benchmarks without requiring any fine-tuning.
What Problem Does This Paper Address?
While foundation models have revolutionized NLP and computer vision, progress in the time series domain has been bottlenecked by data scarcity. Training large-scale Time Series Foundation Models (TSFMs) from scratch is challenging because massive, high-quality, publicly available time series datasets are hard to come by.
To circumvent this, researchers have tried repurposing pretrained language or vision models. However, previous attempts often required computationally expensive fine-tuning, were restricted to narrow tasks (like univariate forecasting), or underperformed compared to specialized TSFMs. This paper addresses a fundamental question: Can we effectively reuse powerful, frozen vision representations for time series classification, and if so, why does it work?
Methods
The TiViT framework consists of a simple yet highly effective pipeline:
- Time Series-to-Image Transformation: A 1D time series is normalized using robust scaling and segmented into patches of length $P = \sqrt{T}$ (where $T$ is the series length). These patches are stacked vertically to form a 2D matrix, which is then replicated across three channels to create a grayscale image and resized to match the standard input resolution of a ViT.
- Feature Extraction via Frozen ViTs: The generated image is fed into a frozen, pretrained ViT backbone (such as OpenCLIP, DINOv2, or SigLIP 2).
- Intermediate Layer Aggregation: Instead of using the final layer's output or the standard
[CLS]token, TiViT extracts embeddings from an intermediate layer (typically between 40% and 70% of the model's total depth) and averages the representations across all tokens. - Classification: A simple linear classifier (logistic regression) is trained on top of these static embeddings.
- TSFM Fusion: To maximize performance, TiViT representations can be directly concatenated with embeddings from specialized time series models before classification.
Datasets
The framework was rigorously evaluated on two major archives:
- UCR Archive: 128 univariate time series datasets spanning various domains (e.g., healthcare, robotics, sensor data).
- UEA Archive: 27 multivariate time series datasets (after filtering out 3 datasets due to short sequence lengths or small test sizes).
For the baseline comparison models, a pretraining corpus of 100,000 samples was curated from diverse public datasets including ECG, EMG, Epilepsy, FD-A/B, Gesture, HAR, and SleepEEG.
Key Insights and Ideas
- The Theoretical Edge of 2D Patching: The authors mathematically prove that reshaping a time series into a 2D grid and applying 2D patching reduces sample complexity during training compared to traditional 1D patching. 2D patching spreads discriminative, label-relevant signals across a larger fraction of tokens, making it significantly easier for the Transformer's attention mechanism to lock onto important patterns.
- The "Sweet Spot" is in the Middle: The final layers of vision models are heavily optimized for high-level visual semantics (e.g., distinguishing a cat from a dog), which destroys the abstract structural properties of a reshaped time series. The authors discovered that intermediate layers retain the highest intrinsic dimensionality—meaning they preserve the most complex, uncompressed structural information, making them ideal for non-visual downstream tasks.
- Strong Cross-Domain Complementarity: Representation space alignment analysis revealed that frozen ViTs and specialized TSFMs (like Mantis) extract fundamentally different features from the same data. Because their representation spaces have very low alignment, they are highly complementary.
Key Results
- Outperforming Specialized TSFMs: On the 128 UCR datasets, TiViT (using an OpenCLIP ViT-H/14 backbone) achieved a mean linear classification accuracy of 81.3%, outperforming the state-of-the-art in-domain TSFM, Mantis (80.1%).
- Better Together (Feature Fusion): Concatenating TiViT features with Mantis features yielded a substantial performance boost, reaching 83.0% accuracy on UCR and 73.7% on UEA.
- Zero-Shot Capability: Evaluated with a simple nearest-centroid classifier (requiring no logistic regression training), TiViT achieved a zero-shot accuracy of 71.6% on the UCR benchmark, demonstrating excellent feature generalizability.
Limitations and Future Work
While TiViT delivers impressive results, the authors note a few limitations:
- Channel Independence: For multivariate datasets, TiViT processes each channel independently and simply concatenates the final embeddings. It does not explicitly model cross-channel correlations.
- Simplistic Fusion: The integration of TSFM and ViT features relies on basic concatenation; more sophisticated multimodal fusion strategies remain unexplored.
- Frozen Backbone: The vision weights are kept completely frozen. Unfreezing or parameter-efficient fine-tuning (PEFT) of the ViT backbone could potentially unlock even higher performance.
Summary updated