Series-wise Cross-validation for Global Time Series Forecasting

Vitor Cerqueira, Mariana Oliveira, Luis RoquearXiv 2026 · arXiv

TL;DR

This paper introduces series-wise cross-validation (CV), a new evaluation framework for global time series models that partitions data across both series and time. The authors demonstrate a crucial trade-off: while traditional time-wise CV is better at estimating the exact magnitude of error, series-wise CV is superior at selecting models that generalize to entirely new, unseen time series.

What Problem Does This Paper Address?

In the past, time series forecasting relied on "local" models like ARIMA, where a separate model was fit to every individual series. Naturally, the standard way to evaluate these was time-wise CV: you train on the past and test on the future of that specific series.

However, modern forecasting has shifted toward global models—large deep learning architectures (like N-BEATS or PatchTST) trained on thousands of different series simultaneously. These models are often deployed in "evolving populations" where new products, sensors, or customers appear after the model is trained. The authors argue that standard time-wise CV fails to account for this. It only tests if a model can extrapolate on known dynamics, not if it can generalize to entirely new series.

Methods

The authors formalize series-wise CV, where the data split happens across the "space" of series as well as the "time" dimension.

Figure 1: Illustration of time-wise CV (a) and series-wise CV (b). In time-wise CV, all series contribute to both training and testing, with only the temporal dimension used for splitting. In series-wise CV, the split is across series and time: one subset is used for training, a disjoint subset for testing, and both are further split in time to preserve temporal order.

They benchmark several strategies for partitioning these series:

  • Holdout: A simple random split (e.g., 70% of series for training, 30% for testing).
  • K-fold CV: Partitioning series into $K$ disjoint blocks and rotating the test set.
  • Monte Carlo CV: Repeatedly drawing random subsets of series for training and testing.
  • Bootstrap: Sampling series with replacement for the training set.
Figure 1: Illustration of time-wise CV (a) and series-wise CV (b). In time-wise CV, all series contribute to both training and testing, with only the temporal dimension used for splitting. In series-wise CV, the split is across series and time: one subset is used for training, a disjoint subset for testing, and both are further split in time to preserve temporal order.

To ensure a fair comparison, they use a nested CV approach. An "inner loop" (time-wise) handles hyperparameter tuning, while the "outer loop" (series-wise or time-wise) evaluates the final performance estimation and model selection quality.

Datasets

The study uses a massive benchmark consisting of 11 datasets totaling 5,776 time series and over 1.5 million observations. These include:

  • Competition data: Tourism, M1, and M3 (monthly and quarterly).
  • Domain-specific data: ECL (electricity), Exchange (currency), Hospital (admissions), Traffic, and Weather.

They evaluated these using 10 deep learning architectures, including popular models like DLinear, N-BEATS, PatchTST, and Temporal Fusion Transformers (TFT).

Key Insights and Ideas

  • Estimation vs. Selection: A key "aha moment" is that the best method for estimating error isn't necessarily the best for selecting the best model. Time-wise CV gives a more accurate "number" for the expected error, but series-wise CV is better at pointing to which model architecture or hyperparameter set will actually perform best in production.
  • Stricter Evaluation: Series-wise CV is a "stricter" test. By forcing the model to predict series it has never seen before, it rewards configurations that learn generalizable patterns rather than just memorizing specific series dynamics.
  • The "New Series" Advantage: Series-wise CV shines specifically when the test set contains series not present in the training set. If the population of series is static (no new series ever appear), the advantage of series-wise CV disappears, and traditional time-wise CV becomes preferable.

Key Results

The authors evaluated the CV procedures across five dimensions. Key quantitative findings include:

  • Downstream Performance: Series-wise CV variants consistently outperformed time-wise CV. For example, Holdout achieved the best average rank (3.545) compared to Time-CV (4.879).
  • Model Selection: Series-wise methods like K-MC-CV and Holdout showed the lowest "regret" (the gap between the selected model and the theoretical best), indicating they are more robust at picking winning models.
  • Error Estimation: Time-CV achieved the lowest Mean Absolute Performance Estimation Error (MAPEE) at 0.109, confirming it is the most reliable for approximating the actual out-of-sample loss magnitude.
  • Overoptimism: Time-wise CV was more prone to underestimating error (overoptimism rate of 0.324) compared to series-wise methods like Kf-CV (0.258).

Limitations and Future Work

  • Computational Cost: Running $K$ folds for global deep learning models is computationally expensive. However, the authors note that even a single-split Holdout (which is cheap) outperformed time-wise CV in downstream accuracy.
  • Simplified Inner Loop: The study used a simplified time-wise split for the inner hyperparameter tuning loop to keep the experiments tractable. Future work could explore using series-wise splits for both loops.
  • Distribution Shift: The current work focuses on the emergence of new series. The authors suggest future research should investigate other forms of distribution shift, such as gradual drift within existing series.

Summary updated

Abstract

We consider the problem of cross-validation (CV) in time series forecasting, with a focus on global forecasting models based on deep learning. The prevailing CV strategy in forecasting is to partition data across time with the last part of each series used for testing. With global models trained on collections of time series, however, a fundamentally different approach becomes possible: partitioning across series and time. We formalize series-wise CV for global forecasting models and benchmark several instantiations, including K-fold CV, bootstrapping, and Monte Carlo CV, on 11 datasets with 10 neural network architectures. We evaluate CV procedures along five dimensions: performance estimation, overoptimism rate, model selection accuracy, regret, and downstream forecasting performance. We withhold a random fraction of series from training so that they appear only in the test set. This protocol simulates the emergence of new series not seen during training, a common scenario in global forecasting. Under this protocol, our results reveal a nuanced trade-off: series-wise CV yields better downstream forecasting accuracy by guiding model selection toward configurations that generalize across both time and series, while time-wise CV retains a slight advantage for estimating out-of-sample error.
BibTeX
@misc{cerqueira2026series,
  author = {Vitor Cerqueira and Mariana Oliveira and Luis Roque},
  title = {{Series-wise Cross-validation for Global Time Series Forecasting}},
  year = {2026},
  eprint = {2611.08252},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08252},
}
Figure