Abstract
With the increasing concern on climate change and global warming, the reduction of carbon emission becomes an important topic in many aspects of human society. The development of energy Internet (EI) makes it possible to achieve better utilization of distributed renewable energy sources with the power sharing functionality introduced by energy routers (ERs). In this paper, a bottom-up EI architecture is designed, and a novel data-driven dynamical control strategy is proposed. Intelligent controllers augmented by deep reinforcement learning (DRL) techniques are adopted for the operation of each microgrid independently in the bottom layer. Moreover, the concept of curriculum learning (CL) is integrated into DRL to improve the sample efficiency and accelerate the training process. Based on the power exchange plan determined in the bottom layer, considering the stochastic nature of electricity price in the future power market, the optimal power dispatching scheme in the upper layer is decided via model predictive control. The simulation has shown that, under the bottom-up architecture, compared with the conventional methods such as proportional integral and optimal power flow, the proposed method reduces overall generation cost by 7.1% and 37%, respectively. Meanwhile, the introduced CL-based training strategy can significantly speed up the convergence during the training of DRL. Last but not least, our method increases the profit of energy trading between ERs and the main grid.
Introduction
As the global energy landscape transitions toward renewable sources, the "Energy Internet" (EI) has emerged as a vision for a highly controllable, information-led energy system. However, traditional energy management typically follows a top-down approach, where a central controller solves massive global optimization problems. This model faces significant challenges in scalability, data privacy, and the ability to handle the stochastic nature of renewable generation and market prices.
To address these issues, this research proposes a shift toward a bottom-up architecture. Instead of a centralized command, individual microgrids (MGs) are empowered to decide their local resource management independently, sharing only necessary power exchange data with the upper grid layers.
System Architecture
The proposed framework organizes the Energy Internet into a two-tier hierarchy within a microgrid cluster, as illustrated in the architecture below.
- The Bottom Layer (Microgrids): Each microgrid acts as an autonomous agent managing local assets such as solar photovoltaics (PV), battery energy storage (BES), fuel cells (FC), and local loads.
- The Upper Layer (Energy Routers): Energy Routers (ERs) facilitate power sharing between microgrids and coordinate with the main grid. This layer optimizes energy flow based on the aggregated needs of the bottom layer.
Methodology
The control strategy integrates advanced machine learning with classical control theory to manage uncertainty and complexity.
Local Control via Deep Reinforcement Learning
Each microgrid utilizes the Advantage Actor-Critic (A2C) DRL algorithm. This allows agents to learn optimal control policies for local assets through interaction with the environment, effectively handling the volatility of solar power and demand.
Curriculum Learning (CL)
To overcome the "data-hungry" nature of standard DRL, the authors implemented Curriculum Learning. By training a policy on one microgrid and transferring that knowledge to others, the system provides a "near-optimal" starting point for new agents, drastically improving sample efficiency.
Upper Layer Optimization
Once microgrids determine their power exchange requirements, the upper layer employs Model Predictive Control (MPC). This layer uses Geometric Brownian Motion to model future electricity price fluctuations, allowing the system to make smarter, more profitable trading decisions with the main grid.
Experimental Results
The system was validated using real-world PV and load data from Dataport (Pecan Street Inc.). Key findings include:
- Cost Efficiency: The bottom-up DRL approach reduced overall generation costs by 7.1% compared to Proportional-Integral (PI) control and by 37% compared to conventional Optimal Power Flow (OPF) methods.
- Training Speedup: Curriculum Learning enabled agents to reach near-optimal performance almost immediately, whereas vanilla DRL required approximately 1 million episodes to converge.
- Profitability: The method increased energy trading profits between energy routers and the main grid by a factor of 11 (from 1.5 to 16.5 units) by optimizing trades against market volatility.
- Hardware Longevity: Unlike OPF, which can cause battery states to oscillate, the DRL approach produced smoother control signals, potentially extending battery life.
Conclusion and Future Work
The bottom-up architecture provides a scalable and privacy-preserving solution for the Energy Internet. By decentralizing decision-making and utilizing transfer learning, the system achieves superior economic performance and training efficiency. Future work will focus on dynamic weight tuning for multi-objective optimization to better balance conflicting priorities between homeowners and utility providers.
Reference: Hua, H., Qin, Z., et al. "Data-driven Dynamical Control for Bottom-up Energy Internet System." Journal of LaTeX Class Files, Vol. 14, No. 8, August 2015 (Preprint).
Summary updated