TSDS-Toolbox: A Toolbox for Measuring Time-Series Dataset Similarity

Yen-Ku Liu, Hongjie Chen, Ryan A. Rossi, Franck DernoncourtarXiv 2026 · arXiv

TL;DR

TSDS-Toolbox is a new open-source, unified framework designed to measure and benchmark similarity between entire time-series datasets. By standardizing data preprocessing, similarity computation, and downstream task evaluation, it helps researchers reliably select the best source datasets for fine-tuning time-series foundation models.

What Problem Does This Paper Address?

With the rise of time-series foundation models, transfer learning and fine-tuning have become crucial for tasks like forecasting and anomaly detection. To do this effectively, engineers need to find a source dataset that closely matches their target domain.

However, existing tools primarily focus on measuring similarity between individual time-series sequences rather than entire datasets. The few methods that do compute dataset-level similarity are implemented in a fragmented, inconsistent manner, making fair comparison nearly impossible. Researchers are often forced to build evaluation pipelines from scratch, leading to inconsistent experimental setups. TSDS-Toolbox solves this by providing a modular, configuration-driven benchmarking pipeline.

Methods

The TSDS-Toolbox architecture is split into four distinct layers:

Figure 1: Overview of the TSDS-Toolbox architecture and its modules.
  1. Data Layer: Loads heterogeneous formats (CSV, GluonTS, DataFrames) and applies uniform preprocessing like train-test splitting and normalization.
  2. Similarity Layer: Supports two main paradigms for calculating distance matrices:
  3. Dataset-Level Metrics: Treats datasets as empirical distributions and compares them directly (e.g., Wasserstein Distance, Maximum Mean Discrepancy (MMD), Optimal Transport, and Match-and-Deform).
  4. Series-Level Metrics (Reducer-based): Compresses a dataset into a single representative sequence using a "reducer" (like DTW Barycenter Averaging (DBA) or PCA) and then computes standard sequence distances (DTW, Euclidean Distance, or Longest Common Subsequence).
  5. Evaluation Layer: Tests how well these similarity scores correlate with actual performance on downstream tasks like classification (via k-NN out-of-distribution membership) and forecasting (using foundation models like Lag-Llama and Time-MoE).
  6. Analysis Layer: Generates visual and quantitative reports, including heatmaps, network graphs, and Pearson correlation metrics.

The toolbox also maps out specific constraints for each metric, such as whether they support multivariate data or varying sequence lengths:

Table 1: Supported similarity methods and their constraints in TSDS-Toolbox. ✓/✗denote support/non-support under the default setting; 𝑁and 𝑇denote the number of time series and sequence length. ∗: multivariate support after flattening; †: Euclidean distance requires equal length by default but supports truncate and resample policies.

Datasets

The authors evaluated the toolbox using 25 GluonTS datasets spanning various real-world domains, including traffic, weather, electricity, exchange rates, tourism, and public health. For consistency, each dataset was sampled with replacement to obtain 100 windows, each with a sequence length of 100.

Key Insights and Ideas

  • Dataset similarity is highly task-dependent: A metric that successfully predicts which dataset will yield the best classification performance might fail entirely when applied to a forecasting task. There is no "one-size-fits-all" similarity metric.
  • Dataset-level metrics shine in forecasting: Advanced distributional metrics like Match-and-Deform (MAD) and Wasserstein Distance (WSD) are particularly effective at capturing both distribution shifts and temporal deformations, making them highly predictive of out-of-distribution forecasting performance.
  • Reducers matter: When using series-level metrics, the choice of reducer is critical. DTW Barycenter Averaging (DBA) generally preserves temporal dynamics better than PCA, leading to stronger task correlations in most settings.

Key Results

The authors used Pearson correlation to measure how well dataset similarity scores align with downstream task losses (where a higher correlation means the similarity metric is a better predictor of performance):

Table 2: Correlation between dataset similarity and down- stream task loss. Cls. denotes classification; OOD-Lag and OOD-TMoE denote forecasting OOD evaluation with Lag- Llama and Time-MoE, respectively; TL-TMoE denotes fore- casting transfer learning with Time-MoE. The symbol “–” indicates that no reducer is used.
  • Forecasting OOD: For out-of-distribution forecasting using the Time-MoE foundation model, Match-and-Deform (MAD) achieved the highest correlation (0.619), closely followed by Wasserstein Distance (0.604).
  • Classification: Optimal Transport (OT) without any reducer proved to be the best predictor for classification tasks, yielding a correlation of 0.417.
  • Visualizations: The toolbox automatically maps out how different metrics interpret dataset relationships, revealing distinct structural clusters depending on the chosen algorithm.
Figure 2: Visualization results generated by TSDS-Toolbox.

Limitations and Future Work

While the paper demonstrates the utility of TSDS-Toolbox, the authors note that reducer effectiveness is highly sensitive to both the downstream task and the underlying series-level distance metric. The paper concludes by positioning TSDS-Toolbox as an extensible foundation, implying that future work will involve integrating a wider variety of time-series foundation models, expanding downstream task types, and developing more robust dataset-level metrics that natively handle multivariate data without flattening.

Summary updated

Abstract

The rapid advancement of artificial intelligence (AI) has significantly accelerated research in time-series analysis, particularly in forecasting, classification, and generation tasks. Recent models, especially foundation models, benefit from time-series dataset similarity due to its significant role in source dataset selection for fine-tuning. However, many existing implementations for benchmarking time-series dataset similarity methods are fragmented and difficult to extend. To address this, we present a unified framework, the Time-Series Dataset Similarity Toolbox (TSDS-Toolbox). Our work enables (1) systematic and reproducible comparisons of time-series dataset similarity methods; (2) flexible extensibility for users to add customized datasets, similarity methods, and downstream time-series tasks; and (3) consistent evaluation of both dataset-level and series-level similarity methods through integrated time-series dataset reducers. The effectiveness of TSDS-Toolbox is validated through comprehensive experiments under diverse experimental settings. Our toolbox is publicly available.
BibTeX
@misc{liu2026tsds,
  author = {Yen-Ku Liu and Hongjie Chen and Ryan A. Rossi and Franck Dernoncourt},
  title = {{TSDS-Toolbox: A Toolbox for Measuring Time-Series Dataset Similarity}},
  year = {2026},
  eprint = {2608.08119},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.08119},
}
Figure