Abstract
While Foundation Models have revolutionized Natural Language Processing and Computer Vision, time series analysis has historically relied on domain-specific models. This research, conducted by Visa Research and UC Riverside, explores the feasibility of a universal time series foundation model. By repurposing the UCR Archive for multi-domain self-supervised pre-training, the authors demonstrate that a single model can learn generalizable features across diverse domains—ranging from medical ECGs to power consumption—resulting in improved classification accuracy and significantly more stable convergence during fine-tuning.
Introduction: The Foundation Model Gap
In the current AI landscape, "Foundation Models" like GPT for text or CLIP for images represent the gold standard. These models are pre-trained on massive, heterogeneous datasets and subsequently fine-tuned for specific downstream tasks. However, time series analysis has struggled to follow this trajectory.
Most time series models are siloed within specific domains, such as finance or medicine, because data varies wildly in frequency, scale, and morphology across different fields. This variance makes it difficult for models to transfer knowledge from one domain to another. This paper addresses a fundamental question: Is it possible to build a single model trained on a diverse "mish-mash" of time series data that effectively solves new, unseen problems?
Methodology: Architectures and Pre-training
To identify the most effective configuration for a time series foundation model, the researchers evaluated 20 distinct combinations of neural network architectures and self-supervised learning (SSL) methods.
Backbones (Architectures)
- Recurrent Neural Networks (LSTM & GRU): Traditional architectures for sequential data.
- ResNet: A robust 1D convolutional neural network (CNN) known for its performance in signal processing.
- Transformer (XFMR): The architecture underpinning modern Large Language Models, utilizing attention mechanisms to capture long-range dependencies within a sequence.
Pre-training Frameworks
The study compared four established SSL methods—SimCLR, TS2Vec, MixingUp, and TF-C—against a novel approach developed by the authors: TimeCLR.
TimeCLR: Optimizing Self-Supervised Learning
TimeCLR builds upon the SimCLR framework but introduces two critical refinements tailored for time series data:
- Diverse Augmentations: The method utilizes a wide array of distortions, including jittering, smoothing, warping, and masking. This variety forces the model to learn features that remain invariant to noise or temporal shifts.
- The "Single Augmentation" Strategy: Unlike many SSL methods that apply multiple simultaneous distortions—often rendering the time series unrecognizable—TimeCLR applies only one random augmentation per positive pair. This ensures the augmented signal retains enough structural integrity to provide a meaningful learning signal.
Data Strategy: Repurposing the UCR Archive
The researchers utilized the UCR Archive, a benchmark collection of 128 diverse time series datasets. To simulate the conditions required for a foundation model, they pooled the training and test sets of all 128 datasets.
- The Split: 50% of this combined pool was used for unlabeled pre-training. The remaining 50% was reserved for 128 individual downstream classification tasks, split into training, validation, and test sets.
- Domain Diversity: The resulting pre-training set included data from food spectrographs, industrial sensors, and medical devices, providing a truly multi-domain training environment.
Key Insights and Results
1. The Power of Multi-Domain Pre-training
The study confirmed that pre-training on a diverse mixture of domains is highly effective. Knowledge gained from power consumption data, for instance, can improve the classification of heartbeats more effectively than training a model from scratch.
2. Stability and Convergence
One of the most significant benefits of the foundation model approach is the stability of the fine-tuning process. Pre-trained models exhibited much smoother convergence curves compared to models trained from scratch. On "hard" datasets where models typically struggle to learn, the foundation model provided a significantly steadier path to high performance.
3. Architecture Performance
While ResNets are often considered superior to RNNs in time series tasks, multi-domain pre-training significantly closed the performance gap, making LSTMs and GRUs competitive again. However, the Transformer architecture emerged as the clear winner, proving most capable of absorbing and utilizing the diverse knowledge present in the multi-domain pre-training set.
4. Overall Superiority
The combination of the Transformer architecture and the TimeCLR method achieved the best overall results. This configuration matched or outperformed the second-best method in approximately 93% of the 128 downstream tasks and consistently held the lowest (best) average rank across all datasets.
Limitations and Future Directions
While this research marks a significant step toward a universal time series model, several challenges remain:
- Dimensionality: The current study focused on univariate (1D) time series. Future research should explore 2D ResNets or architectures specifically designed for multivariate data.
- Scaling: Although the UCR Archive is diverse, it is small compared to the datasets used in NLP. Scaling to "in-the-wild" datasets with billions of data points is the next logical step.
- Task Generalization: This evaluation focused on classification. Future work will determine if these foundation models offer similar performance boosts for forecasting and anomaly detection.
Summary updated