Partially Linear Component Support Vector Machine for Primary Energy Consumption Forecasting of the Electric Power Sector in the United States

Xin Ma, Yubin Cai, Yuan Hong, Yanqiao DengSustainability 2023 · DOI

TL;DR

This paper introduces a novel "Partially Linear Component Support Vector Machine" (PLC-SVM) for more accurate energy consumption forecasting. By combining Principal Component Analysis (PCA) with Support Vector Machines (SVM), PLC-SVM effectively captures both linear and non-linear patterns in time-series data, outperforming existing models in mid- to long-term forecasting of U.S. electric power sector energy consumption.

What Problem Does This Paper Address?

Accurate energy forecasting is crucial for economic planning and environmental management. Traditional time-series forecasting methods often struggle to capture the complex interplay of linear and non-linear dynamics present in energy consumption data. While machine learning models have advanced forecasting capabilities, there's a need for models that can efficiently leverage both linear relationships and complex non-linear patterns without excessive computational cost or hyperparameter tuning. The authors identify a research gap in applying partially linear models, which can theoretically better handle data with mixed linear and non-linear characteristics, to energy forecasting.

Methods

The core innovation is the Partially Linear Component Support Vector Machine (PLC-SVM). Here's how it works:

  • Principal Component Analysis (PCA): Before feeding data into the SVM, PCA is used to reduce the dimensionality of the input features. This step extracts the most significant linear components from the data, effectively creating a lower-dimensional representation that is less prone to multicollinearity and simplifies the linear part of the model.
  • Partially Linear Model Formulation: The model assumes that the target variable (energy consumption) can be represented as a sum of a linear function of the principal components (from PCA) and a non-linear function of the original input features. This is a departure from standard SVMs that typically model the entire relationship as non-linear.
  • Support Vector Machine (SVM) Core: The non-linear component is handled by the kernel trick, a standard SVM technique. The authors use the primal-dual formulation of SVM, which allows for convex optimization and guarantees global convergence during training.
  • Sequential Minimum Optimization (SMO): This algorithm is employed to efficiently solve the dual problem of the PLC-SVM, enabling practical training of the model.
  • Auto-Regressive Forecasting Scheme: For time-series forecasting, the PLC-SVM is integrated into a partially linear auto-regressive framework. This means the model predicts future values based on past values of the time series, with the input to the PLC-SVM being the lagged time-series data.

Datasets

The study utilizes a publicly available dataset of monthly primary energy consumption for the electric power sector in the United States. This dataset spans from January 1973 to January 2020, containing 565 data points. The data was preprocessed by dividing by the largest value in the in-sample data before training and then scaling the predictions back.

Key Insights and Ideas

  • Leveraging Linear Structure: The PLC-SVM's key advantage lies in its explicit modeling of a linear component derived from PCA. This allows it to better capture and utilize inherent linear trends in the data, which standard non-linear models might overlook or struggle to represent efficiently.
  • Dimensionality Reduction for Stability: Using PCA to extract principal linear components not only reduces computational complexity but also enhances the stability of the linear part of the model, preventing issues like multicollinearity.
  • Improved Performance with Longer Lags: The study found that increasing the time lag (the number of past data points used for prediction) generally improved the performance of the PLC-SVM, suggesting that longer historical data provides richer information for the model.
  • Simplicity in Hyperparameter Tuning: Compared to many other complex machine learning models (like tree-based methods or GRUs), PLC-SVM requires tuning fewer hyperparameters, making it more practical for real-world applications.
  • Global Convergence: The use of the primal-dual formulation and SMO ensures that the PLC-SVM training process converges to a global optimum, leading to more reliable and consistent model performance.

Key Results

The PLC-SVM consistently outperformed eight other commonly used machine learning models and a linear auto-regressive model across various evaluation metrics and different time lags (τ=18, τ=24, τ=30).

  • Overall Superiority: In all three case studies, the PLC-SVM achieved the best performance across most of the 19 evaluation metrics.
  • Comparison with SVM: The standard SVM model showed the closest performance to PLC-SVM, highlighting the benefit of the kernel-based approach. However, PLC-SVM consistently surpassed SVM, especially with longer time lags.
  • Tree-Based Models: Models like XGBoost showed competitive performance, sometimes rivaling PLC-SVM, especially in high-dimensional scenarios, but generally fell short of PLC-SVM's overall accuracy.
  • GRU Performance: The GRU model showed promise, particularly with longer lags, but ultimately did not match PLC-SVM's overall performance.
  • Mid-/Long-Term Forecasting: The paper emphasizes that PLC-SVM is particularly well-suited for mid- to long-term forecasting, demonstrating its potential for practical decision-making.

Limitations and Future Work

The authors acknowledge two primary limitations:

  • Small Datasets: The model might not be suitable for datasets with very few data points, as the effectiveness of PCA diminishes with limited data, and consequently, the linear component might not be well-defined.
  • Kernel Choice: The study primarily focused on the Gaussian kernel. Future work could explore other advanced kernels or design new kernels tailored for specific energy forecasting problems to potentially further enhance performance.

The authors suggest that future research could investigate the application of PLC-SVM to forecasting other types of energy and explore more advanced kernel functions. They also note that the model's performance improves with larger time lags, suggesting that incorporating more historical data could be beneficial.

Summary updated

Abstract

Energy forecasting based on univariate time series has long been a challenge in energy engineering and has become one of the most popular tasks in data analytics. In order to take advantage of the characteristics of observed data, a partially linear model is proposed based on principal component analysis and support vector machine methods. The principal linear components of the input with lower dimensions are used as the linear part, while the nonlinear part is expressed by the kernel function. The primal-dual method is used to construct the convex optimization problem for the proposed model, and the sequential minimization optimization algorithm is used to train the model with global convergence. The univariate forecasting scheme is designed to forecast the primary energy consumption of the electric power sector of the United States using real-world data sets ranging from January 1973 to January 2020, and the model is compared with eight commonly used machine learning models as well as the linear auto-regressive model. Comprehensive comparisons with multiple evaluation criteria (including 19 metrics) show that the proposed model outperforms all other models in all scenarios of mid-/long-term forecasting, indicating its high potential in primary energy consumption forecasting.
BibTeX
@inproceedings{ma2023partially,
  author = {Xin Ma and Yubin Cai and Yuan Hong and Yanqiao Deng},
  title = {{Partially Linear Component Support Vector Machine for Primary Energy Consumption Forecasting of the Electric Power Sector in the United States}},
  booktitle = {Sustainability},
  year = {2023},
  doi = {10.3390/su15097086},
}
Figure