Integrating the Expected Future in Load Forecasts with Contextually Enhanced Transformer Models

Raffael Theiler, Olga Fink, Sansavini, Giovanni, Howland, Michael F., Fink, OlgaarXiv 2024 · DOI · arXiv

Abstract

Accurate and reliable energy forecasting is essential for power grid operators who strive to minimize extreme forecasting errors that pose significant operational challenges and incur high intra-day trading costs. Incorporating planning information -- such as anticipated user behavior, scheduled events or timetables -- provides substantial contextual information to enhance forecast accuracy and reduce the occurrence of large forecasting errors. Existing approaches, however, lack the flexibility to effectively integrate both dynamic, forward-looking contextual inputs and historical data. In this work, we conceptualize forecasting as a combined forecasting-regression task, formulated as a sequence-to-sequence prediction problem, and introduce contextually-enhanced transformer models designed to leverage all contextual information effectively. We demonstrate the effectiveness of our approach through a primary case study on nationwide railway energy consumption forecasting, where integrating contextual information into transformer models, particularly timetable data, resulted in a significant average mean absolute error reduction of 26.6%. An auxiliary case study on building energy forecasting, leveraging planned office occupancy data, further illustrates the generalizability of our method, showing an average reduction of 56.3% in mean absolute error. Compared to other state-of-the-art methods, our approach consistently outperforms existing models, underscoring the value of context-aware deep learning techniques in energy forecasting applications.

Introduction

In modern power grid management, forecasting errors carry significant financial and operational risks. Grid operators must maintain a precise balance between supply and demand; underestimation necessitates expensive emergency energy procurement, while overestimation leads to inefficient resource waste. As energy systems transition toward decentralized models—incorporating local solar generation, electric vehicles, and smart buildings—load patterns exhibit increased volatility.

Traditional autoregressive models often struggle with "atypical" events, such as holidays or sudden schedule changes, because they rely primarily on historical patterns. However, information about these events is often available in advance through planning data like train timetables or office schedules. This paper addresses the challenge of effectively integrating this "expected future" contextual information into state-of-the-art forecasting models.

Methodology: Contextually Enhanced Transformers

The core innovation of this work is the reframing of energy load forecasting as a combined forecasting-regression task. Rather than simply looking backward, the proposed architecture treats the future as a sequence of known contextual signals that can be regressed upon.

The researchers developed "Contextually Enhanced" (CE) variants of three prominent Transformer architectures: Crossformer, Spacetimeformer, and a Time-Series Transformer (TST). These models utilize non-causal attention in the decoder, allowing them to process the entire window of future contextual information (FCI)—such as weather forecasts and schedules—simultaneously.

Conceptual framework of the contextually-enhanced forecasting approach, showing the integration of distributed energy resources and the transformer-based architecture.

Key architectural features include:

  • Dual-Stream Embedding: An encoder processes historical load and past context, while the decoder is specifically fed future contextual signals.
  • Simplified Input Layers: Linear embeddings were found to be more effective for future context than complex patching strategies, preventing the "dilution" of strong future signals.

Experimental Evaluation

The models were evaluated on two primary datasets:

  1. Nationwide Railway (SBB): A dataset from the Swiss national railway (2018–2023) containing grid load paired with passenger/freight timetables and train counts.
  2. Building Energy (AlphaBuilding): A simulated office building dataset where future context includes planned occupancy and weather.

Additionally, the models were tested on standard benchmarks (ETTh1 & ETTh2) to ensure general forecasting competitiveness.

Results and Discussion

The integration of future context yielded dramatic improvements in accuracy and reliability. In the railway case study, the CE-Spacetimeformer reduced Mean Absolute Error (MAE) by 26.6% on average. Even more critically for grid stability, extreme outliers (errors > 30%) were reduced by 87.8%.

NMAE performance for Railway and Building Energy test sets, illustrating the impact of adding weather and planning/occupancy data across various models.

In the building energy auxiliary study, the CE-Crossformer achieved a 56.3% reduction in MAE by leveraging planned occupancy data, with a 93.0% reduction in outliers. These results demonstrate that when rich future context is available, the attention mechanism of Transformers is exceptionally well-suited to mapping complex relationships between schedules and energy demand, outperforming simpler linear models like DLinear.

Conclusion

This research underscores the value of context-aware deep learning in energy applications. By treating forecasting as a dual task that incorporates the "expected future," the proposed models nearly eliminate the catastrophic errors that typically plague power grid operations during atypical events. Future work will explore probabilistic forecasting to better manage uncertainty and risk in real-world grid operations.

Summary updated

Abstract

Accurate and reliable energy forecasting is essential for power grid operators who strive to minimize extreme forecasting errors that pose significant operational challenges and incur high intra-day trading costs. Incorporating planning information -- such as anticipated user behavior, scheduled events or timetables -- provides substantial contextual information to enhance forecast accuracy and reduce the occurrence of large forecasting errors. Existing approaches, however, lack the flexibility to effectively integrate both dynamic, forward-looking contextual inputs and historical data. In this work, we conceptualize forecasting as a combined forecasting-regression task, formulated as a sequence-to-sequence prediction problem, and introduce contextually-enhanced transformer models designed to leverage all contextual information effectively. We demonstrate the effectiveness of our approach through a primary case study on nationwide railway energy consumption forecasting, where integrating contextual information into transformer models, particularly timetable data, resulted in a significant average mean absolute error reduction of 26.6%. An auxiliary case study on building energy forecasting, leveraging planned office occupancy data, further illustrates the generalizability of our method, showing an average reduction of 56.3% in mean absolute error. Compared to other state-of-the-art methods, our approach consistently outperforms existing models, underscoring the value of context-aware deep learning techniques in energy forecasting applications.
BibTeX
@misc{theiler2024integrating,
  author = {Raffael Theiler and Olga Fink and Sansavini, Giovanni and Howland, Michael F. and Fink, Olga},
  title = {{Integrating the Expected Future in Load Forecasts with Contextually Enhanced Transformer Models}},
  year = {2024},
  eprint = {2409.05884},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.05884},
}
Figure