Forecast with forecasts: Diversity matters

Yanfei Kang, Wei Cao, Fotios Petropoulos, Feng LiEur. J. Oper. Res 2022 · DOI

TL;DR

This paper proposes a novel approach to forecast combination that leverages the diversity of individual forecasts as a key feature, rather than relying on historical time series characteristics. This "forecasts with forecasts" method simplifies the process, is more robust to limited historical data, and achieves superior forecasting performance for both point and interval predictions.

What Problem Does This Paper Address?

Combining forecasts from multiple models is a well-established technique to improve prediction accuracy. However, a persistent challenge, known as the "forecast combination puzzle," is that simple averages often perform as well as or better than sophisticated methods that try to optimally weight individual forecasts. This is partly because optimal weights are sensitive to historical data and difficult to generalize. Furthermore, existing "feature-based" forecasting methods rely on extracting features from historical data, which can be problematic when data is scarce or when patterns change over time. This paper addresses these limitations by proposing a method that uses the diversity of out-of-sample forecasts themselves as the primary feature for determining combination weights.

Methods

The core of the proposed method is to use pairwise diversity measures between forecasts from different models as features. Specifically, they calculate the squared difference between forecasts from pairs of models for each time step and then aggregate these differences. This forms a "diversity vector" for each time series.

This diversity vector is then used as input to a meta-learning model (specifically, XGBoost) to learn optimal combination weights. The process involves two phases:

  1. Model Training: A reference set of time series is used. For each series, forecasts are generated by a pool of individual methods. Diversity is extracted from these forecasts, and forecasting errors are calculated. An XGBoost model is trained to minimize a cost function that balances point and interval forecast errors, using diversity as the primary input feature.
  2. Forecasting: For a new time series, forecasts are generated by the same pool of methods. The diversity vector is extracted from these forecasts, and the pre-trained XGBoost model is used to predict the optimal combination weights for this new series. These weights are then used to combine the individual forecasts.

This approach contrasts with prior feature-based methods that extract features from historical data. The proposed method focuses on the output of the forecasting models, making it more robust to data limitations and changing patterns.

Datasets

The primary dataset used for evaluation is the M4 data set. This is a large collection of 100,000 time series with diverse seasonal periods and domains, making it a comprehensive benchmark for forecasting methods. The authors also conducted a case study using sales data for fast-moving consumer goods (FMCG) from a North American food manufacturer, comprising 955 unique SKU × location combinations.

Key Insights and Ideas

  • Diversity as a Feature: The paper demonstrates that forecast diversity, measured from out-of-sample forecasts, is a powerful and sufficient feature for determining optimal forecast combination weights. This shifts the focus from historical data characteristics to the behavior of the forecasts themselves.
  • Simplicity and Robustness: By using diversity as the primary feature, the method avoids the complex and often subjective process of selecting historical time series features. This makes the approach simpler, more interpretable, and more robust, especially for time series with limited historical data.
  • Addressing the "Forecast Combination Puzzle": The diversity-based approach offers a way to move beyond simple averaging by intelligently weighting forecasts, effectively tackling the "forecast combination puzzle."
  • Complementary Features: While diversity alone performs well, combining it with traditional time-series features (as in the "FD" approach) can further enhance performance, suggesting that these two types of information are complementary.

Key Results

  • The proposed "Diversity" method achieved performance comparable to or better than the FFORMA approach (which uses 42 historical time series features) in terms of both point forecast accuracy (MASE) and prediction interval quality (MSIS) on the M4 dataset.
  • The Diversity approach outperformed a simple average (SA) by 18.71% in MASE and 19.92% in MSIS, demonstrating the effectiveness of unequal weighting.
  • In the case study on FMCG data, the Diversity method also outperformed FFORMA and SA, highlighting its effectiveness even with a smaller reference dataset.
  • The trade-off curves showed that the Diversity approach offered a very competitive trade-off between upper coverage (service level) and scaled upper prediction intervals (holding cost) across various frequencies.

Limitations and Future Work

  • Model Specificity: The current study focused on linear statistical methods. While the framework is generalizable, future work could explore its application with nonlinear models or a mix of statistical and judgmental forecasts.
  • Probability Density Forecasting: The authors suggest extending the method to probability density forecasting, where diversity could be measured using probability distribution distances (e.g., Kullback-Liebler divergence).
  • Excluding Heterogeneous Forecasts: The current study uses all available forecasts. Future research could explore excluding the most heterogeneous forecasts per series to potentially improve performance.
  • Comparison with Original M4 Contestants: The empirical results on the M4 data are not directly comparable to the original M4 competition contestants due to differences in data access and hyper-parameter tuning. Future work could involve comparing the framework against other established benchmarks on new datasets.

Summary updated

BibTeX
@article{kang2022forecast,
  author = {Yanfei Kang and Wei Cao and Fotios Petropoulos and Feng Li},
  title = {{Forecast with forecasts: Diversity matters}},
  journal = {Eur. J. Oper. Res.},
  year = {2022},
  doi = {10.1016/j.ejor.2021.10.024},
}
Figure