Abstract
This paper investigates the phenomenon of dataset selection bias in time series forecasting research. By evaluating 13 diverse models across 13 benchmark datasets, the authors demonstrate that evaluating on small subsets of data (typically 3–6 datasets) allows a disproportionately large number of models to claim "State-of-the-Art" (SOTA) status. The study reveals that with just four datasets, nearly half of all tested models can be reported as the top performer. These findings highlight a critical need for more rigorous, large-scale evaluation standards to move beyond the "illusion of progress" in the field.
Introduction
In the rapidly evolving landscape of machine learning, new time series forecasting models frequently claim to outperform existing benchmarks. However, the field suffers from a significant reproducibility and generalization problem. Many researchers evaluate their models on a very small number of datasets—often between three and six. This practice leads to "cherry-picking," whether intentional or accidental, where models are shown to excel on a "lucky" slice of the data landscape while their broader weaknesses remain hidden.
Methodology
The researchers developed a systematic framework to quantify the impact of dataset selection on model rankings. Their approach involved:
- Comprehensive Benchmarking: 13 forecasting models (ranging from classical statistical methods like ARIMA and ETS to modern deep learning architectures like Informer and TiDE) were evaluated across 13 benchmark datasets, totaling over 129,000 individual time series and 62 million observations.
- Combinatorial Analysis: Instead of relying on overall averages, the authors simulated cherry-picking by calculating model rankings across every possible subset of datasets for various subset sizes ($n$).
- Stability Metrics: They introduced "Rank Consistency" and "Top-$k$ Datasets" analysis to observe how frequently a model's rank shifts when the evaluation data changes.
The "Best in Class" Fallacy
The quantitative findings serve as a wake-up call for the community. When a researcher selects just four datasets to report, approximately 46% of the models tested could be presented as the "best" model, and 77% could rank in the top three.
As shown in the analysis of subset sizes, the risk of incorrectly identifying a "best" algorithm drops by approximately 40% simply by increasing the number of test datasets from three to six. This demonstrates that SOTA status is often a byproduct of selection bias rather than inherent model superiority.
Model Stability: Classical vs. Deep Learning
A key insight from the study is the variance in performance stability across different architectures. Modern deep learning models, such as Informer, TCN, and DeepAR, show high sensitivity to dataset selection. Their performance varies wildly depending on the specific data used for evaluation.
In contrast, traditional methods like ETS (Exponential Smoothing) and NHITS emerged as the most consistent top performers when looking at the entire data collection. While they may not be #1 in every single subset, their rankings are significantly more stable across different datasets compared to their deep learning counterparts.
Conclusion
The paper serves as a plea for the research community to move beyond "SOTA-chasing" on small data slices. The authors suggest that future work should focus on developing statistical tests for dataset representativeness and creating well-defined benchmark suites that guarantee a diverse range of data characteristics. Ultimately, honest progress in time series forecasting requires large-scale, transparent evaluation to ensure that new models truly generalize to the messy reality of global data.
Summary updated