Research on Dynamic Energy Management Optimization of Park Integrated Energy System Based on Deep Reinforcement Learning

Xinjian Jiang, Lei Zhang, Fuwang Li, Zhi‐Ru Li, Zhijian Ling, Zhenghui ZhaoEnergies 2025 · DOI

TL;DR

This paper tackles the challenge of optimizing energy management in park-level integrated energy systems (IES) that face significant uncertainties from renewable energy sources and fluctuating loads. The authors propose a novel approach using Deep Reinforcement Learning (DRL), specifically comparing the Deep Deterministic Policy Gradient (DDPG) and Deep Q-Network (DQN) algorithms, to dynamically minimize operating costs. Their findings demonstrate that DDPG offers superior cost reduction and robustness compared to DQN, highlighting DRL's effectiveness in handling complex, uncertain energy systems.

What Problem Does This Paper Address?

Modern integrated energy systems (IES) in industrial parks are increasingly incorporating renewable energy sources like wind and solar. While this is crucial for the energy transition, it introduces significant challenges: the intermittent nature of renewables and the unpredictable fluctuations in energy demand create dual uncertainty. Traditional optimization methods, often relying on historical data or deterministic models, struggle to adapt to these dynamic and complex real-time dispatching requirements. This can lead to inefficient operation, increased costs, and even safety risks. The paper aims to bridge this gap by developing a DRL-based approach that can learn optimal energy management strategies directly from data, adapting to real-time conditions without needing precise predictive models.

Methods

The core of the proposed solution is framing the energy management problem as a Markov Decision Process (MDP). The authors define a comprehensive state space that includes electrical and thermal loads, photovoltaic output, energy storage status, time, and grid interaction power. They also define a continuous action space for controlling various energy generation units (like combined heat and power, gas boilers, electric boilers) and energy storage.

Two prominent DRL algorithms are employed and compared:

  • Deep Q-Network (DQN): This algorithm is adapted for the problem, but it requires discretizing the continuous action space, which can lead to a loss of precision and potentially suboptimal strategies.
  • Deep Deterministic Policy Gradient (DDPG): This algorithm is designed to handle continuous action spaces directly, avoiding the discretization issue faced by DQN. It uses an actor-critic architecture to learn both a policy (how to act) and a value function (how good an action is).

A reward function is designed to incentivize minimizing operating costs while penalizing the inefficient use of renewable energy. The authors then compare the performance of DDPG and DQN in a simulated park IES environment.

Datasets

The paper utilizes synthetically generated data for photovoltaic power generation, electrical load, and thermal load. This data was created using the open-source CREST model, which is known for its ability to simulate realistic stochastic energy demand and generation profiles based on probabilistic human behavior and weather patterns. This approach allows for controlled experimentation and evaluation of the algorithms under various uncertainty scenarios.

Key Insights and Ideas

  • DRL for Dynamic IES Management: DRL, particularly DDPG, is highly effective at managing complex, dynamic IES with inherent uncertainties, outperforming traditional methods that struggle with real-time adaptation.
  • Continuous vs. Discrete Actions: DDPG's ability to handle continuous action spaces directly provides a significant advantage over DQN, which requires discretization. This leads to more precise control and better cost optimization.
  • Robustness to Uncertainty: Both DRL algorithms demonstrate strong adaptability to source-load fluctuations, maintaining cost stability even under significant variations.
  • Photovoltaic Penetration Impact: As the penetration of photovoltaic power increases, the effectiveness of DRL algorithms in optimizing energy storage dispatching can decrease, highlighting the need for advanced grid support mechanisms at very high renewable penetration levels.

Key Results

  • Cost Reduction: The DDPG algorithm achieved an average weekly operating cost of 532,424 yuan, which is 8.6% lower than the DQN algorithm's cost of 578,382 yuan.
  • Improved Robustness: DDPG showed a 19.5% reduction in the standard deviation of weekly costs, indicating better stability and robustness.
  • Adaptability to Fluctuations: Even with 10% to 30% source-load fluctuations, the DQN algorithm maintained cost fluctuations below 4.5%, demonstrating DRL's resilience.
  • Faster Convergence: DDPG converged in approximately 4000 training cycles, while DQN required around 10,000 cycles, making DDPG more suitable for real-time applications.
  • Photovoltaic Impact: Higher photovoltaic penetration rates (e.g., 80%) led to a decrease in the effectiveness of DRL in managing energy storage, resulting in more unstable system behavior.

Limitations and Future Work

  • Simulation-Based: The study relies on simulations rather than experimental validation, which is acknowledged as an exploratory scope.
  • Simplified Model: Network losses are not explicitly modeled, and partial-load efficiencies of components are not considered.
  • One-Way Grid Interaction: The model only considers electricity purchase from the main grid, not selling electricity back.
  • Limited Algorithm Comparison: Only DQN and DDPG were compared; other DRL algorithms could offer further insights.
  • Scalability: While the method can be scaled to larger systems, computational and communication challenges may arise.

Future work suggested includes:

  • Hybrid optimization combining model-based and data-driven methods.
  • Incorporating emission and sustainability metrics.
  • Experimental validation in a real-world park environment.
  • Exploring scenarios where the system sells electricity to the grid.
  • Investigating more advanced DRL algorithms.

Summary updated

Abstract

Under the background of energy transition, the Integrated Energy System (IES) of the park has become a key carrier for enhancing the consumption capacity of renewable energy due to its multi-energy complementary characteristics. However, the high proportion of wind and solar resource access and the fluctuation of diverse loads have led to the system facing dual uncertainty challenges, and traditional optimization methods are difficult to adapt to the dynamic and complex dispatching requirements. To this end, this paper proposes a new dynamic energy management method based on Deep Reinforcement Learning (DRL) and constructs an IES hybrid integer nonlinear programming model including wind power, photovoltaic, combined heat and power generation, and storage of electric heat energy, with the goal of minimizing the operating cost of the system. By expressing the dispatching process as a Markov decision process, a state space covering wind and solar output, multiple loads and energy storage states is defined, a continuous action space for unit output and energy storage control is constructed, and a reward function integrating economic cost and the penalty for renewable energy consumption is designed. The Deep Deterministic Policy Gradient (DDPG) and Deep Q-Network (DQN) algorithms were adopted to achieve policy optimization. This study is based on simulation rather than experimental validation, which aligns with the exploratory scope of this research. The simulation results show that the DDPG algorithm achieves an average weekly operating cost of 532,424 yuan in the continuous action space scheduling, which is 8.6% lower than that of the DQN algorithm, and the standard deviation of the cost is reduced by 19.5%, indicating better robustness. Under the fluctuation of 10% to 30% on the source-load side, the DQN algorithm still maintains a cost fluctuation of less than 4.5%, highlighting the strong adaptability of DRL to uncertain environments. Therefore, this method has significant theoretical and practical value for promoting the intelligent transformation of the energy system.
BibTeX
@inproceedings{jiang2025research,
  author = {Xinjian Jiang and Lei Zhang and Fuwang Li and Zhi‐Ru Li and Zhijian Ling and Zhenghui Zhao},
  title = {{Research on Dynamic Energy Management Optimization of Park Integrated Energy System Based on Deep Reinforcement Learning}},
  booktitle = {Energies},
  year = {2025},
  doi = {10.3390/en18195172},
}
Figure