TL;DR
TSDS-Toolbox is a new open-source, unified framework designed to measure and benchmark similarity between entire time-series datasets. By standardizing data preprocessing, similarity computation, and downstream task evaluation, it helps researchers reliably select the best source datasets for fine-tuning time-series foundation models.
What Problem Does This Paper Address?
With the rise of time-series foundation models, transfer learning and fine-tuning have become crucial for tasks like forecasting and anomaly detection. To do this effectively, engineers need to find a source dataset that closely matches their target domain.
However, existing tools primarily focus on measuring similarity between individual time-series sequences rather than entire datasets. The few methods that do compute dataset-level similarity are implemented in a fragmented, inconsistent manner, making fair comparison nearly impossible. Researchers are often forced to build evaluation pipelines from scratch, leading to inconsistent experimental setups. TSDS-Toolbox solves this by providing a modular, configuration-driven benchmarking pipeline.
Methods
The TSDS-Toolbox architecture is split into four distinct layers:
- Data Layer: Loads heterogeneous formats (CSV, GluonTS, DataFrames) and applies uniform preprocessing like train-test splitting and normalization.
- Similarity Layer: Supports two main paradigms for calculating distance matrices:
- Dataset-Level Metrics: Treats datasets as empirical distributions and compares them directly (e.g., Wasserstein Distance, Maximum Mean Discrepancy (MMD), Optimal Transport, and Match-and-Deform).
- Series-Level Metrics (Reducer-based): Compresses a dataset into a single representative sequence using a "reducer" (like DTW Barycenter Averaging (DBA) or PCA) and then computes standard sequence distances (DTW, Euclidean Distance, or Longest Common Subsequence).
- Evaluation Layer: Tests how well these similarity scores correlate with actual performance on downstream tasks like classification (via k-NN out-of-distribution membership) and forecasting (using foundation models like Lag-Llama and Time-MoE).
- Analysis Layer: Generates visual and quantitative reports, including heatmaps, network graphs, and Pearson correlation metrics.
The toolbox also maps out specific constraints for each metric, such as whether they support multivariate data or varying sequence lengths:
Datasets
The authors evaluated the toolbox using 25 GluonTS datasets spanning various real-world domains, including traffic, weather, electricity, exchange rates, tourism, and public health. For consistency, each dataset was sampled with replacement to obtain 100 windows, each with a sequence length of 100.
Key Insights and Ideas
- Dataset similarity is highly task-dependent: A metric that successfully predicts which dataset will yield the best classification performance might fail entirely when applied to a forecasting task. There is no "one-size-fits-all" similarity metric.
- Dataset-level metrics shine in forecasting: Advanced distributional metrics like Match-and-Deform (MAD) and Wasserstein Distance (WSD) are particularly effective at capturing both distribution shifts and temporal deformations, making them highly predictive of out-of-distribution forecasting performance.
- Reducers matter: When using series-level metrics, the choice of reducer is critical. DTW Barycenter Averaging (DBA) generally preserves temporal dynamics better than PCA, leading to stronger task correlations in most settings.
Key Results
The authors used Pearson correlation to measure how well dataset similarity scores align with downstream task losses (where a higher correlation means the similarity metric is a better predictor of performance):
- Forecasting OOD: For out-of-distribution forecasting using the Time-MoE foundation model, Match-and-Deform (MAD) achieved the highest correlation (0.619), closely followed by Wasserstein Distance (0.604).
- Classification: Optimal Transport (OT) without any reducer proved to be the best predictor for classification tasks, yielding a correlation of 0.417.
- Visualizations: The toolbox automatically maps out how different metrics interpret dataset relationships, revealing distinct structural clusters depending on the chosen algorithm.
Limitations and Future Work
While the paper demonstrates the utility of TSDS-Toolbox, the authors note that reducer effectiveness is highly sensitive to both the downstream task and the underlying series-level distance metric. The paper concludes by positioning TSDS-Toolbox as an extensible foundation, implying that future work will involve integrating a wider variety of time-series foundation models, expanding downstream task types, and developing more robust dataset-level metrics that natively handle multivariate data without flattening.
Summary updated