Abstract
This paper aims at developing a data-driven optimal control strategy for virtual synchronous generator (VSG) in the scenario where no expert knowledge or requirement for system model is available. Firstly, the optimal and adaptive control problem for VSG is transformed into a reinforcement learning task. Specifically, the control variables, i.e., virtual inertia and damping factor, are defined as the actions. Meanwhile, the active power output, angular frequency and its derivative are considered as the observations. Moreover, the reward mechanism is designed based on three preset characteristic functions to quantify the control targets: (1) maintaining the deviation of angular frequency within special limits; (2) preserving well-damped oscillations for both the angular frequency and active power output; (3) obtaining slow frequency drop in the transient process. Next, to maximize the cumulative rewards, a decentralized deep policy gradient algorithm, which features model-free and faster convergence, is developed and employed to find the optimal control policy. With this effort, a data-driven adaptive VSG controller can be obtained. By using the proposed controller, the inverter-based distributed generator can adaptively adjust its control variables based on current observations to fulfill the expected targets in model-free fashion. Finally, simulation results validate the feasibility and effectiveness of the proposed approach.
Introduction
As the global energy landscape transitions from fossil fuels to renewable sources, power grids face a critical technical challenge known as the "inertia gap." Traditional power plants utilize massive spinning turbines that provide physical inertia, naturally resisting sudden changes in grid frequency. In contrast, renewable sources like solar and wind connect via power electronics (inverters) that lack inherent inertia. This reduction in system inertia makes the grid more susceptible to frequency instability and potential blackouts.
To address this, researchers have developed Virtual Synchronous Generators (VSGs)—software-controlled inverters designed to mimic the stabilizing behavior of traditional turbines. However, tuning VSG parameters like virtual inertia and damping is complex, as optimal settings vary with grid configuration and load. Traditional rule-based or model-based optimization methods often fail due to their rigidity or the requirement for a perfect mathematical model of the grid.
Proposed Methodology
This research proposes a model-free, data-driven control strategy using Deep Reinforcement Learning (DRL). The VSG tuning problem is framed as a reinforcement learning task where the controller acts as an agent learning to stabilize the grid through trial and error.
The DRL Framework
- Agent: The VSG controller.
- Actions: Real-time adjustment of virtual inertia ($\tilde{H}$) and the damping factor ($\tilde{D}$).
- Observations (State): Active power output, angular frequency, and the rate of change of frequency (ROCOF).
- Reward Function: A multi-objective function that penalizes frequency deviations, rewards well-damped oscillations, and encourages a slow frequency drop during transients.
To accelerate learning, the authors implemented a Decentralized Deep Deterministic Policy Gradient (DDPG) algorithm. By utilizing multiple parallel computation units that exchange information, the system achieves faster convergence compared to standard centralized DRL methods.
Experimental Setup
The proposed approach was validated using a modified IEEE 14-bus test system, a standard benchmark in power systems engineering. The simulation environment included two traditional synchronous generators and one 2.5 MW inverter-based generator (IBDG).
During training, disturbances such as sudden load changes were randomized to ensure the AI developed a robust, generalized policy rather than simply memorizing specific scenarios.
Key Results and Discussion
The decentralized DDPG approach demonstrated significant improvements over traditional and standard DRL methods:
- Efficiency: The decentralized training process took approximately 31.4 hours, a 54% reduction compared to the 68.3 hours required by standard DPG algorithms.
- Stability: In response to a 0.7 MW load disturbance, the AI-controlled VSG maintained frequency deviations within a tight 0.42 Hz limit and effectively eliminated the "ringing" oscillations typical of fixed-parameter controllers.
- Generalization: The controller exhibited strong performance even when tested on different grid layouts, suggesting it learned fundamental principles of grid stability rather than environment-specific rules.
Conclusion
By transforming VSG control into a reinforcement learning task, this research provides a "plug-and-play" solution for stabilizing modern power grids. The model-free nature of the DDPG-based controller allows it to adapt to unknown system dynamics without requiring expert knowledge or complex mathematical modeling. Future work will focus on integrating voltage stability and reactive power control to further enhance grid resilience.
Summary updated