TL;DR
This paper introduces "ESPENI," a novel method to construct a half-hourly electrical demand timeseries for Great Britain by combining publicly available generation and interconnector data. This approach provides a more accurate and granular representation of demand than existing datasets, crucial for understanding and modeling the decarbonization of the UK's energy system.
What Problem Does This Paper Address?
Accurately tracking electrical demand at a granular level is essential for understanding energy system dynamics, especially during periods of rapid change like the decarbonization of Great Britain's electricity sector. Existing publicly available datasets, primarily from Elexon, focus on transmission-connected generation and lack crucial components like distributed generation (solar, wind) and accurate interconnector flows. This omission leads to a significant underestimation of actual demand, hindering detailed analysis of energy transitions. The authors aim to bridge this gap by creating a more comprehensive electrical demand dataset.
Methods
The core of the method involves combining two primary data sources: Elexon's transmission-connected generation data and National Grid's historic demand data.
- Data Integration: Elexon's data, which lacks ISO 8601 compatible datetimes and distributed generation, is combined with National Grid's data, which includes embedded solar and wind generation and more accurate interconnector flow data (including negative values for exports).
- Datetime Standardization: Both datasets are processed to create ISO 8601 compatible UTC and local time datetimes, significantly improving interoperability with other time-series data.
- Data Cleaning and Imputation: The raw data undergoes visual error detection. Identified erroneous rows are flagged, and their values are removed. Missing values are then imputed using simple linear interpolation.
- ESPENI Calculation: The final "ESPENI" (Elexon Sum Plus Embedded Net Imports) timeseries is calculated by summing Elexon's generation data, National Grid's embedded generation data, and net interconnector imports.
Datasets
- Elexon Data: Publicly available half-hourly electricity generation data by fuel type for transmission-connected generators.
- National Grid Data: Publicly available historic demand data, including embedded solar and wind generation and interconnector flows.
- Master Datetime Dataset: A custom dataset created to provide ISO 8601 compatible UTC and local datetimes for mapping to the Elexon and National Grid data.
The paper introduces the ESPENI dataset (both raw and cleaned versions) as a new resource for analyzing Great Britain's electrical demand.
Key Insights and Ideas
- Distributed Generation is Crucial: The inclusion of embedded solar and wind generation from National Grid is vital for accurately approximating total electrical demand, highlighting the growing importance of these sources.
- Interconnector Flows Matter: Properly accounting for net interconnector imports (including exports as negative values) is essential for distinguishing between generation and actual demand.
- Data Granularity Drives Insight: The move from monthly to half-hourly data significantly enhances the ability to analyze dynamic changes in the energy system, such as the rapid decarbonization of coal generation.
- Public Data Value: The authors emphasize the significant value of making such processed data publicly available, saving researchers considerable time and effort.
Key Results
The ESPENI dataset consistently tracks official monthly BEIS electrical demand data more closely than Elexon's generation data alone. Annual ESPENI values were, on average, within 4% of BEIS values, while Elexon's values were significantly lower (around -12.2% in 2020). This demonstrates the improved accuracy of the ESPENI method. The authors also note that the raw Elexon data, while generally high quality, contained a small percentage of visually detected errors (0.15% flagged in the Elexon data, 0% in National Grid data).
Limitations and Future Work
- Error Detection Scalability: The current visual error detection method is not repeatable or scalable, especially for finer granularities (e.g., 5-minute data). The authors suggest that future work should focus on developing robust and repeatable automated methods for error detection and imputation.
- Data Availability: While Elexon and National Grid data are publicly available, they require registration or specific access methods. The authors advocate for more open data licenses and sunset clauses to make historical data a public good.
- Future Work: The authors propose extending the methodology to 5-minute granularity and analyzing the link between electrical demand, supply, and weather patterns. Further research could also explore combining ESPENI data with gas import data to analyze import dependency and investigate the role of HVDC interconnectors in integrating low-carbon generation.
Summary updated