Electricity Price Forecasting based on Order Books: a differentiable optimization approach

Léonard Tschora, Tias Guns, Erwan Pierre, Marc Plantevit, Céline RobardetDSAA 2023 · OpenReview

Abstract

We consider day-ahead electricity price forecasting on the European market. In this market, participants can offer electricity for sale or purchase for a specific price by submitting overnight orders. Market operators determine the market clearing price – the price at which the amount of electricity supplied equals the amount of electricity demanded – using the Euphemia balancing algorithm. EUPHEMIA is a quadratic optimization problem that maximizes the social welfare defined as the sum of the supplier surplus and consumer surplus while ensuring a null energy balance. This mechanism deeply influences the price calculation, but has so far been little considered in electricity price forecasting algorithms. Existing models are generally based on identifying relationships between exogenous characteristics (consumption and production forecasts) and the market clearing price to be predicted. A few studies have examined the EUPHEMIA mechanism during prediction, by doing costly manual transformations on order books. In this article, we overcome this limitation by considering the pricing mechanism during model training. For this, we use a predict-and-optimize strategy with differentiable optimization. We design a fully differentiable and scalable solving method for the EUPHEMIA optimization problem and apply it on real-life data from the European Power Exchange (EPEX). We design different model architectures using our differentiable solver and empirically study the impact of taking into account the optimal calculation of prices within the training of the neural network.

Introduction

Most electricity price forecasting (EPF) models treat the market as a "black box." They typically map exogenous data—such as weather forecasts, expected demand, and fuel prices—directly to a future price. However, European electricity prices are determined by a specific optimization algorithm called EUPHEMIA. Market participants submit "order books" (bids to buy or sell at specific prices), and EUPHEMIA solves a quadratic programming problem to find the "market clearing price"—the point where supply meets demand while maximizing social welfare.

The EUPHEMIA mechanism: Market clearing price is determined by the intersection of supply and demand curves, maximizing social welfare.

Standard machine learning models often ignore this structural knowledge. While some researchers have attempted to include order book data, the process has historically been manually intensive and computationally expensive. This research bridges that gap by making the market-clearing process a differentiable layer within a deep learning model.

Methodology: Predict-and-Optimize

The authors propose a decision-focused learning approach. Instead of just minimizing the error between a predicted price and a real price, the model is trained to minimize the error of the entire decision pipeline.

Differentiable EUPHEMIA Solver

The core innovation is a fully differentiable solving method for the EUPHEMIA optimization problem. While general-purpose differentiable solvers (like OptNet) exist, they are often too slow for large-scale market data. The authors developed a custom solver based on a dichotomy search on the dual problem, which significantly reduces complexity to $O(\log_2(|OB|))$.

Predict-and-Optimize Architecture

The model uses a shared neural network to process exogenous features, which then splits into branches to predict different components:

  • Direct Price Prediction: A standard DNN branch.
  • Order Book Prediction: A branch that predicts the components of the supply and demand order books.
The Predict-and-Optimize (P&O) architecture integrates direct price prediction with order book (OB) prediction and a differentiable solver.

Hybrid Loss Function

The training loss is a weighted combination of three factors: the direct price prediction error ($L_{DNN}$), the order book prediction error ($L_{OB}$), and the "predict-and-optimize" error ($L_{P\&O}$)—the difference between the price resulting from the predicted order book and the actual market price. This allows the model to learn the structural relationships of the market while remaining guided by the final price target.

Experimental Setup

The study utilizes real-world data from the European Power Exchange (EPEX Spot SE) covering France (FR), Germany (DE), Belgium (BE), and the Netherlands (NL) from January 2016 to December 2019. Features include consumption and generation forecasts (including renewables), fuel prices, and historical prices from nine neighboring European countries. To handle the massive size of raw order books, the authors developed a procedure to "shrink" them into a fixed-size representation (20 orders) that preserves the original market-clearing price and volume.

Key Insights and Results

The "Predict-and-Optimize" (P&O) models consistently outperformed standard Deep Neural Networks (DNN) across most metrics (MAE, DAE, RMAE, and SMAPE) for the BE, DE, and NL datasets.

  • Market Sensitivity: The method works best in markets like Germany, Belgium, and the Netherlands, which rely heavily on gas and coal. In these markets, marginal costs are tightly coupled to commodity prices, making order books highly informative.
  • The Nuclear Exception: In France, the model struggled to outperform baselines due to the dominance of nuclear power. Nuclear marginal costs are less sensitive to the exogenous factors the model tracks, making its order books harder to predict.
  • Optimization as a Regularizer: Even a small weight (5%) assigned to the optimization loss during training significantly improved performance, suggesting that incorporating the market mechanism acts as a powerful regularizer.
  • Performance Gains: In Belgium, the hybrid model achieved a Mean Absolute Error (MAE) of 6.28, compared to 7.74 for the standard DNN—a nearly 19% improvement. Qualitative analysis also showed that the P&O model more accurately predicted price spikes during periods of low "residual load."

Conclusion

By integrating the EUPHEMIA market-clearing mechanism into the neural network's training loop via a custom differentiable solver, this framework significantly improves electricity price forecasting accuracy. While currently limited to linear orders, future work will explore more complex constraints, such as block orders, to further align the model with real-world market complexities.

Summary updated

Abstract

We consider day-ahead electricity price forecasting on the European market. In this market, participants can offer electricity for sale or purchase for a specific price by submitting overnight orders. Market operators determine the market clearing price – the price at which the amount of electricity supplied equals the amount of electricity demanded – using the Euphemia balancing algorithm. EUPHEMIA is a quadratic optimization problem that maximizes the social welfare defined as the sum of the supplier surplus and consumer surplus while ensuring a null energy balance. This mechanism deeply influences the price calculation, but has so far been little considered in electricity price forecasting algorithms. Existing models are generally based on identifying relationships between exogenous characteristics (consumption and production forecasts) and the market clearing price to be predicted. A few studies have examined the EUPHEMIA mechanism during prediction, by doing costly manual transformations on order books. In this article, we overcome this limitation by considering the pricing mechanism during model training. For this, we use a predict-and-optimize strategy with differentiable optimization. We design a fully differentiable and scalable solving method for the EUPHEMIA optimization problem and apply it on real-life data from the European Power Exchange (EPEX). We design different model architectures using our differentiable solver and empirically study the impact of taking into account the optimal calculation of prices within the training of the neural network.
BibTeX
@inproceedings{tschora2023electricity,
  author = {Léonard Tschora and Tias Guns and Erwan Pierre and Marc Plantevit and Céline Robardet},
  title = {{Electricity Price Forecasting based on Order Books: a differentiable optimization approach}},
  booktitle = {DSAA},
  year = {2023},
  url = {https://openreview.net/forum?id=PSnKJSnZXW},
}
Figure