Abstract
Time series analysis (TSA) has long struggled with data scarcity and domain-specific silos, hindering the development of robust foundation models similar to those in NLP and Computer Vision. This paper presents SymTime, a foundation model pre-trained on a massive synthetic dataset of 40 million series-symbol pairs. By leveraging the theoretical connection between dynamical systems and symbolic expressions, SymTime treats mathematical formulas as the "DNA" of time series. Through a multi-modal architecture and contrastive learning, the model aligns numerical patterns with their symbolic representations. SymTime achieves state-of-the-art performance across forecasting, classification, imputation, and anomaly detection, demonstrating that synthetic data can effectively bridge the gap to general-purpose time series intelligence.
Introduction: The Data Scarcity Challenge
In the current landscape of Artificial Intelligence, foundation models like GPT-4 and CLIP have revolutionized their respective fields by training on trillions of data points. However, Time Series Analysis has yet to experience its "ImageNet moment." Real-world time series data is often limited in scale, fragmented across private sectors (e.g., healthcare, finance), and plagued by class imbalance. These factors prevent models from generalizing to out-of-distribution scenarios.
To address this, the authors propose a "synthetic-first" approach. Instead of relying solely on scarce real-world data, they ask whether we can generate infinite, high-quality training data by modeling the underlying mathematical principles of dynamics.
Methodology
Series-Symbol ($S^2$) Generation
The core innovation lies in the Series-Symbol ($S^2$) generation mechanism. Inspired by Takens’ Theorem and symbolic dynamics, the authors view time series as observable outputs of complex dynamical systems. The generation process involves:
- Expression Tree Construction: Randomly building mathematical trees using various operators (e.g.,
sin,exp,+,-). - Input Sampling: Generating base sequences from mixed distributions and ARMA (AutoRegressive Moving Average) processes.
- Symbolic Mapping: Passing these inputs through the generated expressions to produce diverse and complex output series.
SymTime Architecture
SymTime is a multi-modal foundation model designed to process both numerical sequences and symbolic formulas. Its architecture consists of two primary components:
- Time Series Encoder: A Transformer-based encoder that utilizes Masked Time Series Modeling (MTM) to learn the structural properties of sequences by predicting missing segments.
- Symbolic Encoder: A DistilBERT-based model that treats mathematical formulas as text, capturing the semantic essence of the underlying dynamics.
The "secret sauce" of SymTime is Contrastive Learning. By using a contrastive loss function, the model aligns the representations from both encoders. This forces the time series encoder to recognize the semantic meaning of patterns—for instance, identifying that a specific oscillating curve corresponds to a "decaying sine wave."
Experimental Setup and Datasets
The model was pre-trained on the synthetic $S^2$ dataset, which contains 40 million series-symbol pairs (approximately 50 billion data points). To evaluate its generalizability, the authors tested SymTime on a wide array of real-world benchmarks:
- Forecasting: ETT, Weather, Electricity, Traffic, and M4.
- Classification: 10 diverse datasets from the UEA multivariate benchmark.
- Anomaly Detection: Five major datasets including SMD and SWaT.
- Imputation: ETT, Weather, and ECL.
Key Insights and Results
SymTime's "synthetic-first" strategy yields several critical insights:
- Scaling Laws: Performance on real-world tasks improves consistently as the size of the synthetic pre-training data increases, suggesting that synthetic data is a viable substitute for real-world data.
- State-of-the-Art Performance: SymTime rivals or exceeds the performance of large-scale models like Time-LLM and Moirai across all five major TSA tasks. Notably, it achieved a top average F1-score of 86.31% in anomaly detection.
- Semantic Understanding: t-SNE visualizations indicate that the model naturally clusters time series based on their mathematical properties (e.g., grouping trigonometric patterns), proving it has moved beyond simple statistical memorization.
- Efficiency: Despite its high performance, SymTime maintains a smaller parameter footprint and lower memory requirements than many LLM-based competitors.
Limitations and Future Work
The authors identify a few areas for improvement. The current generation algorithm can produce overly complex trees that are difficult to sample. Additionally, differential (ODE/PDE) and integral operators are currently excluded to avoid computational instability. Future research will focus on scaling the model and integrating these complex physics-based simulations to further enhance the diversity of the synthetic training environment.
Code available at: https://github.com/wwhenxuan/SymTime
Summary updated