Calculating Great Britain's half-hourly electrical demand from publicly available data

Grant Wilson, Shivangi Sharma, Joseph Day, Noah GodfreyEnergy Strategy Reviews 2022 · OpenReview

TL;DR

This paper introduces a method to create a half-hourly electrical demand dataset for Great Britain, named ESPENI, by combining publicly available generation and interconnector data. This new dataset offers a more comprehensive view of demand than existing sources, crucial for understanding the UK's energy transition and for future energy system modeling.

What Problem Does This Paper Address?

Accurately tracking a nation's electrical demand is fundamental for managing its energy system, especially during periods of significant change like decarbonization. While data on electricity generation is often available, it typically doesn't capture all components of demand, particularly distributed generation (like rooftop solar) and net imports/exports. Existing datasets often lack standardized time formats, making them difficult to integrate and analyze. This paper addresses the challenge of creating a granular, accurate, and interoperable dataset that approximates Great Britain's electrical demand, filling a critical gap for researchers and policymakers tracking the energy transition.

Methods

The core of the method involves combining two primary data sources: Elexon's transmission-level generation data and National Grid's historic demand data. Key steps include:

  • Data Integration: Merging Elexon's generation data (from power plants connected to the high-voltage transmission grid) with National Grid's data that includes embedded/distributed generation (solar and wind) and interconnector flows.
  • Datetime Standardization: Parsing raw data, which lacks ISO 8601 compatible datetimes, and creating standardized UTC and local time values. This is achieved by using a separate master datetime dataset.
  • Data Cleaning: Visually identifying and flagging data errors in the combined dataset.
  • Error Imputation: Using simple linear interpolation to fill in missing values caused by flagged errors.
  • Demand Approximation: Calculating the final "ESPENI" (Elexon Sum Plus Embedded Net Imports) value by summing all generation sources and adding net imports (or subtracting net exports).

Datasets

  • Elexon Data: Publicly available half-hourly generation data from transmission-connected power plants.
  • National Grid Data: Publicly available historic demand data, including embedded solar and wind generation, and interconnector flows.
  • Master Datetime Dataset: A custom dataset used to provide ISO 8601 compatible datetimes for both Elexon and National Grid data.

The paper also references the BEIS (Department for Business, Energy and Industrial Strategy) monthly electrical demand data as a benchmark for comparison.

Key Insights and Ideas

  • The Importance of Distributed Generation and Interconnectors: The paper highlights that solely relying on transmission-connected generation data (like Elexon's) significantly underestimates actual demand, especially as embedded solar and wind grow. Including these components, along with net imports/exports, is crucial for a more accurate picture.
  • Data Interoperability is Key: The effort to standardize datetime formats to ISO 8601 is a critical step that makes the data usable for a wider range of analyses and integration with other time-series datasets.
  • Visual Inspection as a Necessary Evil (for now): While the authors acknowledge the limitations of visual error detection (lack of repeatability and scalability), it was deemed necessary for cleaning the data, especially given the complexity and potential for errors in raw public datasets.
  • ESPENI as a Better Proxy for Demand: The ESPENI dataset consistently tracks official monthly BEIS demand data more closely than Elexon data alone, demonstrating the value of the combined approach.

Key Results

  • The developed method successfully combines Elexon and National Grid data to create a half-hourly electrical demand timeseries for Great Britain, termed ESPENI.
  • The ESPENI dataset shows a significantly smaller annual difference (within 4%) compared to official BEIS monthly demand data than the Elexon data alone (which can be up to 12.2% lower).
  • The paper makes the raw and cleaned ESPENI datasets publicly available, saving researchers considerable time in data preparation.

Limitations and Future Work

  • Error Detection Scalability: The current method for error detection relies on visual inspection, which is not repeatable or scalable, especially for higher-granularity data (e.g., 5-minute data).
  • Imputation Method: While linear interpolation was effective for the observed error block lengths, it might be less suitable for longer gaps in data.
  • Future Work: The authors suggest extending the methodology to 5-minute granularity, developing more robust and repeatable automated methods for error detection and imputation, and further analyzing the relationship between electrical demand, supply, and weather patterns. They also propose using the ESPENI dataset for detailed analysis of changes in generation fuel types and interconnector usage.

Summary updated

Abstract

Here we present a method to combine half-hourly publicly available electrical generation and interconnector operational data for Great Britain to create a timeseries that approximates its electrical demand. We term the calculated electrical demand ‘ESPENI’ that is an acronym for Elexon Sum Plus Embedded Net Imports. The method adds value to the original data by combining both transmission and distribution generation data into a single dataset and adding ISO 8601 compatible datetimes to increase interoperability with other timeseries. Data cleansing is undertaken by visually flagging errors and then using simple linear interpolation to impute values to replace the flagged data points. Publishing the method allows it to be further enhanced or adapted and to be considered and critiqued by a wider community. In addition, the published raw and cleaned data is a valuable resource that saves researchers considerable time in repeating the steps presented in the method to prepare the data for further analysis. The data provide a public record of the decarbonisation of Great Britain's electrical system since late 2008, widely seen as an example of rapid decarbonisation of an electrical system away from fossil fuel generation to lower carbon sources.
BibTeX
@article{wilson2022calculating,
  author = {Grant Wilson and Shivangi Sharma and Joseph Day and Noah Godfrey},
  title = {{Calculating Great Britain's half-hourly electrical demand from publicly available data}},
  journal = {Energy Strategy Reviews},
  year = {2022},
  url = {https://openreview.net/forum?id=s8mOa5U8fP},
}
Figure