Abstract
Maintaining the balance between electricity supply and demand is increasingly challenging due to the rise of intermittent renewable energy. This paper introduces a forecasting tool based on Quantile Regression Forests (QRF) designed to predict intra-hour imbalances two hours into the future at 5-minute intervals. Unlike traditional deterministic models, the QRF approach provides full conditional distributions, allowing for the generation of 95% prediction intervals. Testing on Norwegian grid data demonstrates that this method significantly outperforms existing industry benchmarks, providing grid operators with the reliable uncertainty estimates needed for proactive system management.
Introduction: The Balancing Act
Transmission System Operators (TSOs) must ensure that electricity supply perfectly matches demand to maintain a stable system frequency (50 Hz in Europe). Deviations from this balance risk equipment damage or widespread blackouts. The increasing penetration of intermittent renewables like wind and solar, alongside "prosumers," has introduced significant volatility into the grid.
Currently, the Norwegian TSO, Statnett, utilizes a "Planning Table" that approximates imbalances based on hourly market schedules. However, because physical demand fluctuates by the minute while the market operates on an hourly resolution, this tool often misses critical "intra-hour" nuances. Consequently, operators frequently rely on intuition and costly, last-minute balancing reserves.
Methodology: Quantile Regression Forests
The proposed solution utilizes Quantile Regression Forests (QRF) to forecast these imbalances. While standard Random Forests typically provide a single point estimate (the conditional mean), QRFs estimate the full conditional distribution of the target variable. This capability is crucial for grid operators, as knowing the uncertainty (e.g., a 95% prediction interval) is as vital as the forecast itself for risk management.
The forecasting task is structured as a series of univariate problems. To provide a two-hour look-ahead at 5-minute resolution, the researchers trained 24 independent models for each of the five Norwegian market areas, resulting in a total of 120 models. The feature set was kept intentionally "lean," utilizing 24 lagged imbalance values and temporal features such as month, day, hour, minute, and solar elevation.
Data and Patterns
The study leverages a dataset from Statnett spanning 2015–2016, covering five Norwegian pricing areas (NO1–NO5). The data consists of approximately 210,000 points per area at 5-minute intervals.
Patterns in the Chaos
A significant finding was that imbalances exhibit strong, predictable patterns tied to historical data and time. The model successfully identified "shocks" occurring at the start of every hour—points where market-scheduled production shifts abruptly, creating temporary imbalances.
The researchers found that a simplified feature set (past imbalances and time) often outperformed more complex feature engineering, suggesting that the "momentum" of the grid is a primary driver of short-term imbalances.
Key Results
The QRF model consistently outperformed both a Naive benchmark and the proxy for the TSO's Planning Table across all five market areas, achieving lower Mean Squared Error (MSE) and Mean Absolute Error (MAE).
Crucially, the 95% prediction intervals proved highly reliable, with observed imbalances falling within the predicted range approximately 95% of the time. While accuracy naturally decays as the forecast horizon extends toward the two-hour mark, the QRF model maintained its superiority over existing methods throughout the entire window.
Conclusion and Future Directions
While the current QRF model is highly effective, it remains "blind" to certain external factors such as scheduled power plant maintenance, HVDC import/export changes, and direct forward-looking load forecasts. Interestingly, weather data did not provide a significant performance boost for the Norwegian dataset, though it may be more critical in regions with higher solar or wind penetration.
Future work involves deploying a real-time prototype in Statnett’s control room. This phase will focus on gathering qualitative feedback from human operators and integrating "foreseen" features like production plans to further refine the forecast's accuracy and utility.
Summary updated