Synthetic Series-Symbol Data Generation for Time Series Foundation Models

Wenxuan Wang, Kai Wu, Yujian Betterest Li, Dan Wang, Xiaoyu Zhang

Abstract

Time series analysis (TSA) has long struggled with data scarcity and domain-specific silos, hindering the development of robust foundation models similar to those in NLP and Computer Vision. This paper presents SymTime, a foundation model pre-trained on a massive synthetic dataset of 40 million series-symbol pairs. By leveraging the theoretical connection between dynamical systems and symbolic expressions, SymTime treats mathematical formulas as the "DNA" of time series. Through a multi-modal architecture and contrastive learning, the model aligns numerical patterns with their symbolic representations. SymTime achieves state-of-the-art performance across forecasting, classification, imputation, and anomaly detection, demonstrating that synthetic data can effectively bridge the gap to general-purpose time series intelligence.

Introduction: The Data Scarcity Challenge

In the current landscape of Artificial Intelligence, foundation models like GPT-4 and CLIP have revolutionized their respective fields by training on trillions of data points. However, Time Series Analysis has yet to experience its "ImageNet moment." Real-world time series data is often limited in scale, fragmented across private sectors (e.g., healthcare, finance), and plagued by class imbalance. These factors prevent models from generalizing to out-of-distribution scenarios.

To address this, the authors propose a "synthetic-first" approach. Instead of relying solely on scarce real-world data, they ask whether we can generate infinite, high-quality training data by modeling the underlying mathematical principles of dynamics.

Methodology

Series-Symbol ($S^2$) Generation

The core innovation lies in the Series-Symbol ($S^2$) generation mechanism. Inspired by Takens’ Theorem and symbolic dynamics, the authors view time series as observable outputs of complex dynamical systems. The generation process involves:

  1. Expression Tree Construction: Randomly building mathematical trees using various operators (e.g., sin, exp, +, -).
  2. Input Sampling: Generating base sequences from mixed distributions and ARMA (AutoRegressive Moving Average) processes.
  3. Symbolic Mapping: Passing these inputs through the generated expressions to produce diverse and complex output series.

SymTime Architecture

SymTime is a multi-modal foundation model designed to process both numerical sequences and symbolic formulas. Its architecture consists of two primary components:

  • Time Series Encoder: A Transformer-based encoder that utilizes Masked Time Series Modeling (MTM) to learn the structural properties of sequences by predicting missing segments.
  • Symbolic Encoder: A DistilBERT-based model that treats mathematical formulas as text, capturing the semantic essence of the underlying dynamics.

The "secret sauce" of SymTime is Contrastive Learning. By using a contrastive loss function, the model aligns the representations from both encoders. This forces the time series encoder to recognize the semantic meaning of patterns—for instance, identifying that a specific oscillating curve corresponds to a "decaying sine wave."

Experimental Setup and Datasets

The model was pre-trained on the synthetic $S^2$ dataset, which contains 40 million series-symbol pairs (approximately 50 billion data points). To evaluate its generalizability, the authors tested SymTime on a wide array of real-world benchmarks:

  • Forecasting: ETT, Weather, Electricity, Traffic, and M4.
  • Classification: 10 diverse datasets from the UEA multivariate benchmark.
  • Anomaly Detection: Five major datasets including SMD and SWaT.
  • Imputation: ETT, Weather, and ECL.

Key Insights and Results

SymTime's "synthetic-first" strategy yields several critical insights:

  • Scaling Laws: Performance on real-world tasks improves consistently as the size of the synthetic pre-training data increases, suggesting that synthetic data is a viable substitute for real-world data.
  • State-of-the-Art Performance: SymTime rivals or exceeds the performance of large-scale models like Time-LLM and Moirai across all five major TSA tasks. Notably, it achieved a top average F1-score of 86.31% in anomaly detection.
  • Semantic Understanding: t-SNE visualizations indicate that the model naturally clusters time series based on their mathematical properties (e.g., grouping trigonometric patterns), proving it has moved beyond simple statistical memorization.
  • Efficiency: Despite its high performance, SymTime maintains a smaller parameter footprint and lower memory requirements than many LLM-based competitors.

Limitations and Future Work

The authors identify a few areas for improvement. The current generation algorithm can produce overly complex trees that are difficult to sample. Additionally, differential (ODE/PDE) and integral operators are currently excluded to avoid computational instability. Future research will focus on scaling the model and integrating these complex physics-based simulations to further enhance the diversity of the synthetic training environment.

Code available at: https://github.com/wwhenxuan/SymTime

Summary updated

BibTeX
@inproceedings{wang2025synthetic,
  author = {Wenxuan Wang and Kai Wu and Yujian Betterest Li and Dan Wang and Xiaoyu Zhang},
  title = {{Synthetic Series-Symbol Data Generation for Time Series Foundation Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=xB1ZNgq0Xp},
}
Figure