Optimal Operation of a Hydrogen-based Building Multi-Energy System Based on Deep Reinforcement Learning

Liang Yu, Shuqi Qin, Zhanbo Xu, Xiaohong Guan, Chao Shen, Dong YuearXiv 2021 · OpenReview

TL;DR

This paper tackles the challenge of optimizing the operation of hydrogen-based building multi-energy systems (HBMESs) by incorporating building thermal dynamics, which previous methods often ignored. They propose a novel multi-agent deep reinforcement learning algorithm (MADACR) that can learn optimal control strategies without needing prior knowledge of system parameters or explicit building models, leading to significant cost reductions.

What Problem Does This Paper Address?

Modern buildings are significant energy consumers and carbon emitters. Hydrogen-based multi-energy systems (HBMESs) offer a promising path towards cleaner energy by integrating hydrogen storage with other energy sources like solar PV and battery storage. However, optimizing these complex systems is difficult due to several factors:

  • Uncertainty: Renewable energy generation (PV), electricity prices, and outdoor temperatures are inherently variable.
  • Complex Dynamics: Building thermal loads and the interactions between various energy storage systems (hydrogen, thermal, electric) create intricate, temporally coupled constraints.
  • Model Inaccuracy: Developing accurate and efficient models for building thermal dynamics is challenging and time-consuming.

Existing optimization approaches often simplify these problems by neglecting building thermal dynamics, thus failing to leverage the flexibility of thermal loads for cost savings. This paper aims to bridge this gap by developing an intelligent energy management system that accounts for these complexities.

Methods

The core of this paper lies in its innovative approach to solving the complex optimization problem.

  • Problem Formulation: The authors first formulate the problem as an expected operational cost minimization for an HBMES, considering building thermal dynamics and multiple energy storage systems.
  • Markov Game Reformulation: Due to the inherent uncertainties and coupled constraints, the problem is reformulated as a Markov game, a framework suitable for multi-agent decision-making under uncertainty.
  • MADACR Algorithm: A novel algorithm called Multi-Agent Discrete Actor-Critic with Rules (MADACR) is proposed. This algorithm leverages deep reinforcement learning (DRL) with a multi-agent architecture. Key features include:
  • Centralized Training, Decentralized Execution: Agents are trained collaboratively but operate independently during execution.
  • Actor-Critic Framework: Each agent has an actor (to decide actions) and a critic (to evaluate those actions).
  • Discrete Action Space with GS Sampler: To handle the complexity, actions are discretized, and the Gumbel-Softmax sampler is used to enable back-propagation for training.
  • Rule-Based Action Adjustment: Specific rules are applied to ensure that the selected actions are physically feasible and to reduce the exploration space, improving convergence.
  • No Prior Knowledge Required: Crucially, MADACR does not need prior knowledge of uncertain parameters, parameter predictions, or explicit building thermal dynamics models, making it highly practical.

Datasets

The simulations utilize real-world data for:

  • Electricity Price: Retail commercial prices from June 1 to September 30, 2019, in Beijing.
  • Power Load: Daily power demand data.
  • PV Generation: Solar irradiance data from June 1 to September 30, 2019, from NREL Solar Radiation Research Laboratory.
  • Outdoor Temperature: Data from Pecan Street database.

The data from 90 days was used for training, and 30 days for testing. The paper also mentions four baseline methods (B1-B4) for performance comparison, with B4 representing an upper performance bound assuming perfect information.

Key Insights and Ideas

  • Thermal Dynamics are Key: Ignoring building thermal dynamics means missing out on significant opportunities to reduce operational costs by intelligently shifting thermal loads.
  • Multi-Agent RL for Complex Systems: HBMESs are inherently multi-agent problems, and DRL provides a powerful framework to learn coordinated control strategies for these complex, interacting components.
  • MADACR's Practicality: The ability of MADACR to operate without explicit models or parameter predictions is a major advantage, making it applicable to real-world scenarios where such information is often unavailable or difficult to obtain.
  • Rule-Based Refinements Improve Performance: Incorporating specific rules for action adjustment (e.g., prioritizing BESS over HESS for charging/discharging) significantly improves the feasibility and efficiency of the learned policies.
  • Hydrogen's Role in Flexibility: Hydrogen storage, when optimally managed, can provide crucial flexibility, especially in balancing renewable energy generation and meeting demand at different times.

Key Results

The proposed MADACR algorithm demonstrated significant improvements in operational cost reduction:

  • Cost Reduction: The algorithm reduced operational costs by 8.1% to 44.26% compared to baseline methods (B1-B3) across different simulation cases.
  • Comfort Maintained: These cost savings were achieved while maintaining comfortable indoor temperature ranges, with average temperature deviations (ATD) comparable to or better than baselines.
  • Comparison to Ideal Scenario: While not achieving the absolute best performance of Baseline 4 (which assumes perfect information), the proposed algorithm offers a practical and effective solution that significantly outperforms other heuristic and model-based approaches.
  • Robustness: The algorithm showed robustness to varying thermal disturbances, maintaining cost reduction benefits with only a minor sacrifice in ATD.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while the proposed algorithm is more practical than methods requiring perfect information (like B4), it still relies on some assumptions about the environment and agent interactions.
  • The discrete action space, while simplifying the problem, might not capture the full nuance of continuous control.
  • Future Work:
    • Investigate the optimal operation of HBMESs incorporating additional factors like direct hydrogen selling/buying, heat demands, and controllable electric loads.
  • Study the optimal planning problem of HBMESs, considering the investment costs associated with distributed energy sources to minimize overall system expenditure.

Summary updated

Abstract

Since hydrogen has many advantages (e.g., free pollution, extensive sources, convenient storage and transportation), hydrogen-based multi-energy systems (HMESs) have received wide attention. However, existing works on the optimal operation of HMESs neglect building thermal dynamics, which means that the flexibility of building thermal loads can not be utilized for reducing system operation cost. In this paper, we investigate an optimal operation problem of an HMES with the consideration of building thermal dynamics. Specifically, we first formulate an expected operational cost minimization problem related to an HMES. Due to the existence of uncertain parameters, inexplicit building thermal dynamics models, temporally coupled operational constraints related to three kinds of energy storage systems and indoor temperatures, as well as the coupling between electric energy subsystems and thermal energy subsystems, it is challenging to solve the formulated problem. To overcome the challenge, we reformulate the problem as a Markov game and propose an energy management algorithm to solve it based on multi-agent discrete actor-critic with rules (MADACR). Note that the proposed algorithm does not require any prior knowledge of uncertain parameters, parameter prediction, and explicit building thermal dynamics model. Simulation results based on real-world traces show the effectiveness of the proposed algorithm.
BibTeX
@misc{yu2021optimal,
  author = {Liang Yu and Shuqi Qin and Zhanbo Xu and Xiaohong Guan and Chao Shen and Dong Yue},
  title = {{Optimal Operation of a Hydrogen-based Building Multi-Energy System Based on Deep Reinforcement Learning}},
  year = {2021},
  eprint = {2109.10754},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.10754},
}
Figure