Day-Ahead Electricity Demand Forecasting Competition: Post-COVID Paradigm

Mostafa Farrokhabadi, Jethro Browell, Yi Wang, Stephen Makonin, Wencong Su, Hamidreza ZareipourIEEE Open Access Journal of Power and Energy 2022 · DOI

TL;DR

The COVID-19 pandemic shattered traditional electricity demand patterns, rendering many pre-pandemic forecasting models obsolete. This paper summarizes the "IEEE DataPort Day-Ahead Electricity Demand Forecasting Competition," which challenged researchers to develop robust models for this "new normal," ultimately finding that massive multi-model ensembles and daily "online" updates are the keys to navigating extreme volatility.

What Problem Does This Paper Address?

In the power industry, accurate day-ahead load forecasting is the bedrock of grid stability and economic efficiency. Under normal circumstances, demand follows predictable patterns based on weather, time of day, and the day of the week. However, the COVID-19 pandemic acted as a global "black swan" event.

Industrial shutdowns, business closures, and the massive shift to working from home caused global electricity demand to plummet and its "shape" to warp. Traditional models trained on years of historical data suddenly struggled, exposing grid operators to significant technical and financial risks. The authors organized this competition to identify which state-of-the-art techniques could best mitigate these pandemic-related uncertainties and to provide a standardized benchmark for the research community.

Methods

The competition focused on deterministic day-ahead forecasting (16 to 40 hours ahead) of hourly load. While standard approaches like ARIMA, multiple linear regression, and basic Deep Neural Networks (DNNs) were common starting points, the top-performing teams utilized more sophisticated strategies:

  • Multi-Model Ensembles: The winning teams didn't rely on a single "silver bullet" model. Instead, they combined dozens—or even hundreds—of base models. For example, the first-place team used an ensemble of 72 models, while the third-place team used a staggering 674 models.
  • Online Learning & Daily Updates: Unlike previous competitions where data was released in large blocks, this competition simulated an operational environment. Participants received daily updates and were required to submit forecasts every 24 hours. The most successful teams used this recent data to continuously re-train or adapt their models to the shifting "Post-COVID" reality.
  • Feature Engineering: Teams incorporated weather variables (temperature, humidity, wind speed) and engineered features like "holiday-adjusted" training data and exponentially smoothed temperature variables to capture thermal inertia.

Datasets

The competition utilized a newly publicized dataset (now available for reproducibility) consisting of:

  • Source: A metropolitan electric utility.
  • Timeframe: Approximately four years of hourly data, spanning March 18, 2017, to February 16, 2021.
  • Variables: Electricity demand (kW) alongside weather observations and forecasts (air pressure, cloud cover, humidity, temperature, wind direction, and wind speed).
  • Evaluation Period: A 30-day "live" test period from March 15 to April 13, 2021.

Key Insights and Ideas

  • The "Recency" Premium: In a volatile environment, the most recent data is the most valuable. Successful teams prioritized "online" adaptation, proving that a model's ability to learn from yesterday's error is more important than its performance on a five-year-old training set.
  • Ensembles Provide Robustness: The dominance of multi-model approaches (similar to the M5 forecasting competition trends) suggests that ensembling is no longer just an "extra" step but a requirement for top-tier accuracy. Combining diverse models helps cancel out individual biases and errors.
  • The Persistence Benchmark is a High Bar: The organizers used a simple "persistence" benchmark (using the load from the most recent similar day as the forecast). Surprisingly, only 9 out of 20 finishing teams significantly outperformed this simple baseline, highlighting how difficult the "Post-COVID" landscape truly is.
  • Statistical Significance Matters: The paper emphasizes that over short evaluation windows (30 days), small differences in Mean Absolute Error (MAE) might just be "noise." They used the Diebold-Mariano test to prove that while the top team was clearly superior, many of the other top-10 finishers were statistically indistinguishable from one another.

Key Results

  • Winning Performance: The top five teams achieved "skill scores" (improvement over the benchmark) between 20% and 30%.
  • Error Metrics: Forecasts were evaluated using Mean Absolute Error (MAE). The competition demonstrated that even with extreme pandemic-related shifts, state-of-the-art ensembles could maintain operational viability.
  • Participation: The event drew 239 unique registrations from around the globe, though the difficulty of the daily submission requirement meant only 20 teams successfully completed the entire 30-day evaluation.

Limitations and Future Work

The authors acknowledge several areas for further exploration:

  • Computational Cost: While massive ensembles (600+ models) win competitions, they are expensive to maintain and "productize" in real-world control rooms. Future work needs to balance marginal accuracy gains against operational overhead.
  • Generalizability: The models were tested on a single metropolitan utility. It remains to be seen how these specific "Post-COVID" strategies translate to different geographic regions or different scales (e.g., individual building level vs. city-wide).
  • Explainability: As models become more complex (ensembles of ensembles), they become "black boxes." The authors suggest that future competitions should focus not just on accuracy, but on the explainability of the predictions to help grid operators trust the results.

Summary updated

Abstract

The COVID-19 related shutdowns have made significant impacts on the electric grid operation worldwide. The global electrical demand plummeted around the planet in 2020 continuing into 2021. Moreover, demand shape has been profoundly altered as a result of industry shutdowns, business closures, and people working from home. In view of such massive electric demand changes, energy forecasting systems struggle to provide an accurate demand prediction, exposing operators to technical and financial risks, and further reinforcing the adverse economic impacts of the pandemic. In this context, the “IEEE DataPort Day-Ahead Electricity Demand Forecasting Competition: Post-COVID Paradigm” was organized to support the development and dissemination state-of-the-art load forecasting techniques that can mitigate the adverse impact of pandemic-related demand uncertainties. This paper presents the findings of this competition from the technical and organizational perspectives. The competition structure and participation statistics are provided, and the winning methods are summarized. Furthermore, the competition dataset and problem formulation is discussed in detail. Finally, the dataset is published along with this paper for reproducibility and further research.
BibTeX
@article{farrokhabadi2022day,
  author = {Mostafa Farrokhabadi and Jethro Browell and Yi Wang and Stephen Makonin and Wencong Su and Hamidreza Zareipour},
  title = {{Day-Ahead Electricity Demand Forecasting Competition: Post-COVID Paradigm}},
  journal = {IEEE Open Access Journal of Power and Energy},
  year = {2022},
  doi = {10.1109/oajpe.2022.3161101},
}
Figure