Abstract
We argue that the current practice of evaluating AI/ML time-series forecasting models, predominantly on benchmarks characterized by strong, persistent periodicities and seasonalities, obscures real progress by overlooking the performance of efficient classical methods. We demonstrate that these "standard" datasets often exhibit dominant autocorrelation patterns and seasonal cycles that can be effectively captured by simpler linear or statistical models, rendering complex deep learning architectures frequently no more performant than their classical counterparts for these specific data characteristics, and raising questions as to whether any marginal improvements justify the significant increase in computational overhead and model complexity. We call on the community to (I) retire or substantially augment current benchmarks with datasets exhibiting a wider spectrum of non-stationarities, such as structural breaks, time-varying volatility, and concept drift, and less predictable dynamics drawn from diverse real-world domains, and (II) require every deep learning submission to include robust classical and simple baselines, appropriately chosen for the specific characteristics of the downstream tasks' time series. By doing so, we will help ensure that reported gains reflect genuine scientific methodological advances rather than artifacts of benchmark selection favoring models adept at learning repetitive patterns.
Introduction: The Benchmark Echo Chamber
In the field of Long-horizon Time-Series Forecasting (LTSF), the pursuit of "State-of-the-Art" (SOTA) status has increasingly encountered the pitfalls of Goodhart’s Law: "When a measure becomes a target, it ceases to be a good measure." Recent research suggests that the community has entered a benchmark "echo chamber," where complex Transformers and Large Language Models (LLMs) are validated against a narrow set of "standard" datasets dominated by predictable seasonal cycles.
Because patterns like hourly electricity usage or traffic occupancy are so repetitive, sophisticated neural networks often end up overfitting to these cycles rather than learning deep temporal dynamics. This results in researchers chasing marginal decimal-point improvements in error metrics while ignoring the fact that a basic one-layer linear model or a decades-old statistical method could achieve comparable results at a fraction of the computational cost.
Methodology and Comparative Analysis
This position paper employs a meta-analysis of recent literature and statistical evaluation to challenge the prevailing narrative of progress in LTSF. The authors utilize several analytical techniques:
- Visual & Statistical Analysis: Using Autocorrelation Function (ACF) plots, the authors demonstrate that popular benchmarks (such as the ETT datasets) possess extremely high periodicity, making them "trivial-to-model" for simple architectures.
- The "Seeking SOTA Athlete" Thought Experiment: This analogy describes a "generalist" model that appears superior on average across diverse benchmarks but is consistently outperformed by specialized models on individual, critical tasks.
- Comparative Benchmarking: The authors aggregate results from dozens of recent papers to show that classical models (e.g., Auto-ARIMA) and simple linear models (DLinear) remain highly competitive with massive foundation models.
The Reality of "SOTA" Performance
A critical examination of performance metrics reveals a startling trend: classical models frequently produce Mean Squared Error (MSE) scores that are nearly identical to 2024–2025 "SOTA" models.
As seen in the comparative data, on the ETTh1 dataset, a simple Autoregressive (AR) model from 1951 achieves an MSE of 0.358, while many complex Transformer variants hover between 0.360 and 0.400. This suggests that years of architectural innovation have, in some cases, resulted in zero net gain over basic statistics for these specific datasets.
The Need for Taxonomy-Specific Evaluation
The authors argue that the community must move away from "average" performance metrics and toward a taxonomy-based evaluation. Models should be tested on their ability to handle specific temporal characteristics rather than just repetitive cycles.
By distinguishing between patterns like structural breaks, time-varying volatility, and random walks, researchers can identify where deep learning truly adds value. For example, while general AI models claim to forecast weather 30 days out on standard benchmarks, professional meteorological models only manage ~10 days of reliable forecasts on real-world data, highlighting a significant disconnect between academic benchmarks and real-world physics.
Conclusion and Future Directions
To restore rigor to time-series forecasting, the authors propose a "Path to Rigorous TSF Evaluation":
- Mandatory Baselines: Every new deep learning paper must include simple baselines (ARIMA, DLinear) to prove the added complexity is justified.
- Diverse Benchmarking: The community must adopt more challenging datasets that go beyond simple daily/weekly cycles, such as the GIFT-Eval framework.
- Taxonomy-Specific Evaluation: Models should be tested on specific types of data (e.g., handling a sudden market crash) rather than just global averages.
- Transparency: Moving away from "cherry-picked" results toward standardized, live leaderboards.
Ultimately, the goal is to ensure that reported gains reflect genuine scientific methodological advances rather than artifacts of benchmark selection favoring models adept at learning repetitive patterns.
Summary updated