Introduction
The proliferation of time-series foundation models, such as Lag-Llama and PatchTST, has introduced a significant challenge in model selection. While these models are pre-trained on vast, heterogeneous datasets, their performance on a specific, unseen target dataset remains difficult to predict without conducting resource-intensive experiments. Traditional similarity metrics often focus on individual time series (e.g., Dynamic Time Warping) or are tailored for discrete classification tasks, leaving a gap in measuring similarity at the dataset level for continuous data.
This research proposes a distribution-based framework to quantify the similarity between entire time-series datasets. By modeling datasets as Multivariate Normal Distributions (MVNs) and employing the Wasserstein distance, the authors provide a training-free proxy for predicting model generalization.
Methodology
The proposed approach treats a dataset as a collection of $N$ random vectors (windows) of length $L$. The methodology consists of three primary stages:
- Sampling: Random windows are extracted from the underlying distribution of the dataset.
- Modeling: Each dataset is approximated as a Multivariate Normal Distribution (MVN). The mean vector ($\hat{\mu}$) and covariance matrix ($\hat{\Sigma}$) are estimated using maximum likelihood estimation.
- Distance Calculation: The 2-Wasserstein distance is calculated between the MVN representations of two datasets.
A key advantage of the MVN assumption is the existence of a closed-form solution for the 2-Wasserstein distance, which integrates both the Euclidean distance between means and the differences in covariance structures. This ensures computational efficiency, scaling linearly with the number of samples $N$.
Experimental Setup
The study evaluates the metric using 30 diverse time-series datasets from the GluonTS library, spanning domains such as energy, weather, finance, and web traffic. To ensure comparability, all data were min-max normalized to a $[0, 1]$ range. For the experiments, $N=20,000$ windows of length $L=48$ were sampled from each dataset.
Results and Discussion
The Wasserstein distance effectively captures the underlying relationships between datasets. Visualizations using force-directed graphs demonstrate that the metric naturally clusters datasets by domain (e.g., energy-related datasets grouping together) without requiring manual labels. This "map of data" provides an intuitive way to understand the relatedness of diverse time-series sources.
Crucially, the metric exhibits strong predictive power regarding model performance. The authors found a high correlation (typically $> 0.60$) between the Wasserstein distance and the inference loss of foundation models in both Out-of-Distribution (OOD) and Transfer Learning (TL) scenarios.
Comparison with Baselines
The Wasserstein approach outperforms simpler baselines like Euclidean distance or DTW on averaged series. While Euclidean distance only considers mean shifts, the Wasserstein distance captures the "shape" and "spread" of the data distributions through the covariance term, providing much more granular and informative insights into dataset relationships.
Limitations and Future Work
The current model assumes that sampled windows are independent and identically distributed (i.i.d.), which may overlook long-range temporal dependencies inherent in many time series. Additionally, some datasets showed anomalous correlations, possibly due to significant deviations from the MVN assumption or limited relevance of the source data to the backbone model. Future research will explore non-i.i.d. modeling and the application of this metric to meta-learning and data distillation.
Conclusion
Modeling time-series datasets as MVNs and measuring their similarity via Wasserstein distance provides a robust, efficient, and interpretable metric. This approach serves as a valuable tool for researchers to navigate the landscape of time-series foundation models, offering a reliable, training-free estimate of model performance on new data domains.
Summary updated