HOPS: High-order Polynomials with Self-supervised Dimension Reduction for Load Forecasting

Peng Song, Feng Han, S. Shukla, Jue Wang, Tao HongarXiv 2025 · DOI · arXiv

TL;DR

HOPS (High-Order Polynomials with Self-supervised dimension reduction) revives multivariate polynomial models for electric load forecasting by overcoming the "Curse of Dimensionality." By combining low-rank approximations with a fast, GPU-accelerated Conjugate Gradient optimizer, the authors achieve state-of-the-art accuracy while eliminating the need for tedious manual feature engineering.

What Problem Does This Paper Address?

In the world of smart grids, accurate load forecasting is the difference between efficient energy distribution and costly system failures. While researchers have a buffet of models to choose from—ranging from simple Multiple Linear Regression (MLR) to complex Neural Networks—multivariate high-order polynomials have largely been left off the menu.

On paper, polynomials are great: they have solid mathematical foundations and desirable optimization properties. In practice, they suffer from the Curse of Dimensionality. As you increase the order of a polynomial (e.g., moving from squared terms to cubic terms), the number of parameters explodes exponentially. This leads to three major headaches:

  1. Overfitting: Too many parameters allow the model to "memorize" noise in the training data.
  2. Computational Cost: Solving these models becomes prohibitively slow and memory-intensive.
  3. Manual Labor: Forecasters often spend weeks manually crafting "interaction variables" (like Temperature $\times$ Humidity) to help simpler models capture complex relationships.

Methods

The authors introduce HOPS, a framework designed to make high-order polynomials practical and efficient. The approach rests on three pillars:

  1. Self-supervised Dimension Reduction: Before expanding the features into high-order polynomials, HOPS uses a low-rank linear transformation to project the input data into a lower-dimensional space. This is "self-supervised" because it learns the structure of the input features ($X$) without needing the target load values ($y$) initially. This drastically reduces the parameter count from $O(n^d)$ to $O(k^d)$, where $k$ is the much smaller reduced dimension.
  2. High-order Polynomial Expansion: Once the data is in a manageable dimension, the model applies polynomials up to the third order. This allows the model to automatically capture complex interactions between variables that would otherwise need to be manually engineered.
  3. PolyCG Optimizer: To solve the resulting optimization problem, the authors developed a modified Conjugate Gradient algorithm (PolyCG). Unlike standard solvers that might struggle with massive matrices, PolyCG leverages tensor representations, making it highly efficient for parallel computing on modern GPUs using frameworks like PyTorch.

Datasets

The study utilizes 10 hourly load datasets from ISO New England (ISONE), a standard benchmark in the field.

  • Zones covered: Maine, New Hampshire, Vermont, Connecticut, Rhode Island, and several Massachusetts regions.
  • Features: Hourly dry-bulb temperature and dew point (used to calculate relative humidity).
  • Timeframe: Data from 2012–2014 was used for training/validation, and 2015 was used for testing.

Key Insights and Ideas

  • Automated Feature Engineering: The most significant "aha moment" is that high-order polynomials can replace the "expertise-heavy" process of variable construction. HOPS captures intricate relationships (like how humidity amplifies the effect of temperature) mathematically rather than through trial-and-error manual selection.
  • Subspace Equivalence: The authors provide a mathematical proof (Theorem 1) showing that the model's performance is robust. As long as the dimension reduction matrix spans the same subspace, the final forecasting results remain equivalent.
  • Efficiency vs. Accuracy: The paper demonstrates that you don't need a massive number of input variables to get high accuracy. A HOPS model with only 47 simple variables outperformed a standard regression model using 289 complex, manually crafted variables.

Key Results

  • Superior Accuracy: Across all 10 ISONE datasets, HOPS consistently outperformed vanilla MLR and advanced benchmarks. For example, the "ReHOPS" variant achieved an average MAPE (Mean Absolute Percentage Error) of 3.35%, compared to 3.66% for the well-regarded Recency model.
  • Computational Gains: The PolyCG algorithm proved highly effective. Training the HOPS47 model (47 variables) took only 19 minutes and 24 seconds for all 10 datasets, whereas a more complex version took over an hour, with negligible difference in accuracy.
  • Fewer Variables, Better Forecasts: HOPS demonstrated that by using high-order terms on a reduced feature set, it could achieve lower error rates with significantly less "redundant variable construction."

Limitations and Future Work

  • Hyperparameter Sensitivity: While the model is robust, choosing the optimal embedding dimensions ($k$) still requires a grid search on a validation set, which adds a layer of pre-processing time.
  • Normalization Constraints: The authors noted that applying Z-score normalization makes their dimension reduction nearly equivalent to PCA. They chose to avoid it due to the categorical nature of load data, but exploring different normalization schemes for diverse data types remains a future direction.
  • Higher Orders: The experiments capped the polynomial order at $d=3$. Future work could investigate whether even higher orders provide diminishing returns or if they can be stabilized with more aggressive regularization.

Summary updated

Abstract

Load forecasting is a fundamental task in smart grid. Many techniques have been applied to developing load forecasting models. Due to the challenges such as the Curse of Dimensionality, overfitting, and limited computing resources, multivariate higher-order polynomial models have received limited attention in load forecasting, despite their desirable mathematical foundations and optimization properties. In this paper, we propose low rank approximation and self-supervised dimension reduction to address the aforementioned issues. To further improve computational efficiency, we also utilize a fast Conjugate Gradient based algorithm for the proposed polynomial models. Based on the load datasets from the ISO New England, the proposed method high-order polynomials with self-supervised dimension reduction (HOPS) demonstrates higher forecasting accuracy over several competitive models. Additionally, experimental results indicate that our approach alleviates redundant variable construction, achieving better forecasts with fewer input variables.
BibTeX
@misc{song2025hops,
  author = {Peng Song and Feng Han and S. Shukla and Jue Wang and Tao Hong},
  title = {{HOPS: High-order Polynomials with Self-supervised Dimension Reduction for Load Forecasting}},
  year = {2025},
  eprint = {2501.10637},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.10637},
  howpublished = {arXiv},
}
Figure