Evaluation Study of Linear Combination Technique for SVM related Time Series Forecasting

Xian Cheng, Ji Wu, Jin XuProceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences 2019 · DOI

Abstract

Time series forecasting and Support Vector Machines (SVM) are widely utilized in domains such as smart city development and digital services. This paper empirically investigates the performance of eight linear combination techniques using the M3 competition dataset, comprising 3,003 time series. The results reveal that the "forecast combination puzzle" does not exist for SVM-related forecasting models, as the simple average method consistently performs poorly compared to more sophisticated weighting schemes.

Introduction

In time series forecasting, the "forecast combination puzzle" refers to the empirical observation that a simple average (SA) of multiple models often outperforms complex statistical weighting methods. While this phenomenon is well-documented for traditional statistical models like ARIMA, its applicability to modern machine learning models remains under-explored. This study addresses this gap by evaluating linear combination techniques specifically for Support Vector Machine (SVM) ensembles.

Methodology

The study employs a rigorous four-step framework to evaluate forecast aggregation strategies.

Framework for SVM-based time series forecasting and evaluation
  1. Model Generation: A diverse pool of constituent SVM models is generated by varying hyperparameters, including the penalty constant $C$, insensitive loss $\epsilon$, and kernel types (Linear, Polynomial, and Radial Basis Function).
  2. Selection: For each individual time series, the top eight performing SVM models are selected based on their in-sample Mean Squared Error (MSE).
  3. Combination: Eight distinct linear combination techniques are applied to the selected models. These include the Simple Average (SA), rank-based methods (Median, Trimmed Mean, Winsorized Mean, Inverse Rank), variance-based methods (Bates & Granger), and eigenvector-based methods (Standard and Bias-corrected Eigenvectors).
  4. Evaluation: The performance of these combined forecasts is assessed against out-of-sample data using standard metrics: Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Mean Absolute Percentage Error (MAPE).

Data and Preprocessing

The researchers utilized the M3 Competition dataset, the gold standard for forecasting benchmarks. It contains 3,003 time series categorized by domain (Micro, Industry, Macro, Finance, Demographic, and Other) and time interval (Yearly, Quarterly, and Monthly). Standard preprocessing steps, such as log transformation and deseasonalization, were applied to ensure data compatibility with SVM training requirements.

Results and Analysis

The most significant finding of this study is the debunking of the "forecast combination puzzle" in the context of SVMs. Across nearly all datasets and metrics, the Simple Average (SA) was one of the least effective techniques.

Performance comparison of linear combination techniques for Yearly data

As illustrated in the results for Yearly data, the Bias-corrected Eigenvector (BEC) method achieved a substantial improvement in accuracy. BEC recorded an $RMSE_{average}$ of 116.38, whereas the Simple Average resulted in a much higher error of 325.84. For higher-frequency data (Quarterly and Monthly), which often exhibit seasonal trends, the Inverse Rank (INV) and Bates & Granger (BG) methods emerged as the top performers.

Conclusion

This research demonstrates that for AI-based forecasting models like SVMs, sophisticated weighting schemes that account for relative model performance and error correlations significantly outperform simple averaging. Future research will investigate the generalizability of these findings to other architectures, such as Long Short-Term Memory (LSTM) networks, and explore non-linear "meta-learning" approaches for forecast combination.

Summary updated

Abstract

Time series forecasting and SVM are widely used in many domains, for example, smart city and digital services. Focusing on SVM related time series forecasting model, in this paper we empirical investigate the performance of eight linear combination techniques by using M3 competition dataset which includes 3003 time series. The results reveals that the “forecast combination puzzle” is not exist for combining SVM related forecasting model as the simple average is almost the worst combination technique.
BibTeX
@inproceedings{cheng2019evaluation,
  author = {Xian Cheng and Ji Wu and Jin Xu},
  title = {{Evaluation Study of Linear Combination Technique for SVM related Time Series Forecasting}},
  booktitle = {Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences},
  year = {2019},
  doi = {10.24251/hicss.2019.151},
}
Figure