CarbonMonitor-Power near-real-time monitoring of global power generation on hourly to daily scales

Biqing Zhu, Zhu Deng, Xuanren Song, Wenli Zhao, Da Huo, Taochun Sun, Piyu Ke, Duo Cui, Chenxi Lu, Haiwang Zhong, Chaopeng Hong, Jian Qiu, Steven J. Davis, Pierre Gentine, Philippe Ciais, Zhu LiuScientific 2023 · DOI

TL;DR

This paper introduces CarbonMonitor-Power, a novel near-real-time global dataset of power generation at hourly to daily scales, covering 37 countries and eight energy source groups. This dataset fills a critical gap for timely analysis of power system dynamics, enabling better understanding of energy transitions and impacts of global events like the COVID-19 pandemic.

What Problem Does This Paper Address?

Accurate and timely monitoring of global power generation is crucial for understanding energy transitions, managing electricity grids, and informing climate policy. However, existing datasets often suffer from significant time lags (months to years) and lack the high temporal resolution (hourly to daily) needed to capture rapid changes. This makes it difficult to analyze the immediate impacts of events like the COVID-19 pandemic, geopolitical shifts, or the integration of variable renewable energy sources. The paper addresses this by creating a dataset that provides near-real-time insights into global power generation.

Methods

The researchers constructed the CarbonMonitor-Power dataset by collecting raw data from national grid operators for 37 countries. They employed a multi-step process:

  • Data Acquisition: Raw data was gathered at the highest available temporal resolution (from 5-minute to monthly intervals).
  • Data Cleaning and Preprocessing: A robust method involving Interquartile Range (IQR) thresholding was used to identify and filter outliers. Missing values were filled using linear interpolation. Crucially, extreme values were retained if they corresponded to known abrupt social changes (e.g., COVID-19 lockdowns) or natural disasters, acknowledging their impact on power systems.
  • Aggregation and Disaggregation: Data from various sources and energy categories were aggregated into eight standardized groups: coal, gas, oil, nuclear, hydro, solar, wind, and other renewables. For countries where data was not directly available at the desired temporal resolution or source granularity, sophisticated disaggregation and imputation techniques were developed, often leveraging lower-frequency reference datasets (e.g., IEA, BP, IRENA) and country-specific models.

Datasets

The primary dataset introduced is CarbonMonitor-Power, which provides daily and hourly power generation data for 37 countries from January 2016 to June 2022.

Reference datasets used for comparison and disaggregation include:

  • IEA (International Energy Agency) Monthly Electricity Statistics: Used for disaggregating fossil fuel sources and for comparison.
  • BP Statistical Review of World Energy: Used for comparison, particularly for countries not covered by IEA.
  • IRENA (International Renewable Energy Agency) Renewable Energy Statistics: Used for disaggregating renewable energy sources and for comparison.
  • Ember: Used for comparison.
  • National Grid Operators' Data: Specific sources include OpenNEM (Australia), China Electricity Council/CEC/CCTD (China), POSOCO (India), OCCTO (Japan), ENTSO-E (EU27), BMRS/Sheffield Solar (UK), United Power System of Russia (Russia), Eskom (South Africa), EIA (United States), Gobierno De Mexico (Mexico), and Coordinador Eléctrico Nacional (Chile).

Key Insights and Ideas

  • The Power of Near-Real-Time Data: The dataset demonstrates how high-frequency data can reveal dynamic patterns in power generation, influenced by daily activities, weekends, holidays, and extreme events.
  • COVID-19's Disruption: The data clearly shows that the COVID-19 pandemic caused significant disruptions in power generation in some countries (e.g., China, India), leading to temporary shifts towards lower carbon intensity, while having a minimal impact in others (e.g., Australia).
  • Geopolitical Impacts: The study highlights how events like the Russian invasion of Ukraine can lead to rapid shifts in energy mix, such as increased coal reliance in the EU27&UK, underscoring the need for timely data to monitor such changes.
  • Data Harmonization Challenges: The paper details the complexities of harmonizing power generation data from diverse sources with varying formats, temporal resolutions, and energy source categorizations, showcasing innovative methods for disaggregation and imputation.

Key Results

  • The CarbonMonitor-Power dataset covers 37 countries, representing approximately 70% of global power production and 68% of global power-related CO2 emissions.
  • The dataset provides hourly to daily resolution data for eight energy source groups: coal, gas, oil, nuclear, hydro, solar, wind, and other renewables.
  • Validation against reference datasets (IEA, BP) shows good agreement for total generation (R² > 0.95), indicating the dataset's reliability at aggregated levels.
  • The dataset successfully captures diurnal, weekly, and seasonal patterns, as well as the impact of events like the COVID-19 pandemic and the Ukraine war on energy mix.

Limitations and Future Work

  • Spatial Resolution: While the dataset covers 37 countries, it is primarily at the national level, with some sub-national data available. The authors acknowledge that some distributed solar power generation might not be fully captured.
  • Data Gaps and Proxy Data: For countries not covered or with missing data, the researchers plan to use proxy data, including fossil fuel consumption, climate reanalysis data, and profiles from neighboring countries with similar characteristics.
  • Oil and Other Renewables Correlation: The correlation with reference datasets for oil and other renewables is lower (R² = 0.661 and R² = 0.682, respectively), attributed to methodological differences in disaggregating thermal power and accounting for smaller energy sources.
  • Future Work: The team is actively working on extending the dataset's spatial resolution and coverage, incorporating more countries and regions, and improving data accuracy through advanced proxy data methods. They also aim to provide continuously updated data through a web platform.

Summary updated

Abstract

We constructed a frequently updated, near-real-time global power generation dataset: CarbonMonitor-Power since January, 2016 at national levels with near-global coverage and hourly-to-daily time resolution. The data presented here are collected from 37 countries across all continents for eight source groups, including three types of fossil sources (coal, gas, and oil), nuclear energy and four groups of renewable energy sources (solar energy, wind energy, hydro energy and other renewables including biomass, geothermal, etc.). The global near-real-time power dataset shows the dynamics of the global power system, including its hourly, daily, weekly and seasonal patterns as influenced by daily periodical activities, weekends, seasonal cycles, regular and irregular events (i.e., holidays) and extreme events (i.e., the COVID-19 pandemic). The CarbonMonitor-Power dataset reveals that the COVID-19 pandemic caused strong disruptions in some countries (i.e., China and India), leading to a temporary or long-lasting shift to low carbon intensity, while it had only little impact in some other countries (i.e., Australia). This dataset offers a large range of opportunities for power-related scientific research and policy-making.
BibTeX
@inproceedings{zhu2023carbonmonitor,
  author = {Biqing Zhu and Zhu Deng and Xuanren Song and Wenli Zhao and Da Huo and Taochun Sun and Piyu Ke and Duo Cui and Chenxi Lu and Haiwang Zhong and Chaopeng Hong and Jian Qiu and Steven J. Davis and Pierre Gentine and Philippe Ciais and Zhu Liu},
  title = {{CarbonMonitor-Power near-real-time monitoring of global power generation on hourly to daily scales}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41597-023-02094-2},
}
Figure