Abstract
Time series forecasting and Support Vector Machines (SVM) are widely utilized in domains such as smart city development and digital services. This paper empirically investigates the performance of eight linear combination techniques using the M3 competition dataset, comprising 3,003 time series. The results reveal that the "forecast combination puzzle" does not exist for SVM-related forecasting models, as the simple average method consistently performs poorly compared to more sophisticated weighting schemes.
Introduction
In time series forecasting, the "forecast combination puzzle" refers to the empirical observation that a simple average (SA) of multiple models often outperforms complex statistical weighting methods. While this phenomenon is well-documented for traditional statistical models like ARIMA, its applicability to modern machine learning models remains under-explored. This study addresses this gap by evaluating linear combination techniques specifically for Support Vector Machine (SVM) ensembles.
Methodology
The study employs a rigorous four-step framework to evaluate forecast aggregation strategies.
- Model Generation: A diverse pool of constituent SVM models is generated by varying hyperparameters, including the penalty constant $C$, insensitive loss $\epsilon$, and kernel types (Linear, Polynomial, and Radial Basis Function).
- Selection: For each individual time series, the top eight performing SVM models are selected based on their in-sample Mean Squared Error (MSE).
- Combination: Eight distinct linear combination techniques are applied to the selected models. These include the Simple Average (SA), rank-based methods (Median, Trimmed Mean, Winsorized Mean, Inverse Rank), variance-based methods (Bates & Granger), and eigenvector-based methods (Standard and Bias-corrected Eigenvectors).
- Evaluation: The performance of these combined forecasts is assessed against out-of-sample data using standard metrics: Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Mean Absolute Percentage Error (MAPE).
Data and Preprocessing
The researchers utilized the M3 Competition dataset, the gold standard for forecasting benchmarks. It contains 3,003 time series categorized by domain (Micro, Industry, Macro, Finance, Demographic, and Other) and time interval (Yearly, Quarterly, and Monthly). Standard preprocessing steps, such as log transformation and deseasonalization, were applied to ensure data compatibility with SVM training requirements.
Results and Analysis
The most significant finding of this study is the debunking of the "forecast combination puzzle" in the context of SVMs. Across nearly all datasets and metrics, the Simple Average (SA) was one of the least effective techniques.
As illustrated in the results for Yearly data, the Bias-corrected Eigenvector (BEC) method achieved a substantial improvement in accuracy. BEC recorded an $RMSE_{average}$ of 116.38, whereas the Simple Average resulted in a much higher error of 325.84. For higher-frequency data (Quarterly and Monthly), which often exhibit seasonal trends, the Inverse Rank (INV) and Bates & Granger (BG) methods emerged as the top performers.
Conclusion
This research demonstrates that for AI-based forecasting models like SVMs, sophisticated weighting schemes that account for relative model performance and error correlations significantly outperform simple averaging. Future research will investigate the generalizability of these findings to other architectures, such as Long Short-Term Memory (LSTM) networks, and explore non-linear "meta-learning" approaches for forecast combination.
Summary updated