Fair Reinforcement Learning Algorithm for PV Active Control in LV Distribution Networks

Maurizio Vassallo, Amina Benzerga, Alireza Bahmanyar, Damien ErnstarXiv 2024 · DOI · arXiv

Introduction

As residential photovoltaic (PV) penetration reaches record levels, low-voltage (LV) distribution networks face significant operational challenges. A primary concern is "reverse power flow": during periods of high solar generation and low local demand, excess electricity flows back toward the transformer, causing voltage levels to exceed safe limits. To maintain grid stability, Distribution System Operators (DSOs) employ Smart Inverters (SIs) to perform Active Power Curtailment (APC), effectively capping solar production.

However, standard APC strategies often lead to geographical inequity. Due to the physics of power lines, customers at the end of a feeder typically experience the highest voltage rises and are consequently curtailed more frequently and aggressively than those closer to the transformer. This creates a financial disparity among neighbors. This paper presents a Deep Reinforcement Learning (DRL) approach designed to balance grid stability with social fairness by distributing the burden of curtailment more equitably.

Methodology: The DDPG Framework

The proposed solution utilizes the Deep Deterministic Policy Gradient (DDPG) algorithm, an actor-critic DRL framework capable of handling the continuous action spaces required for precise inverter control.

The Actor-Critic architecture of the DDPG agent, showing the neural networks used to map grid states to inverter actions.

The agent's "brain" consists of two main components:

  1. The Actor ($\phi$): Learns a policy to map the current grid state (voltages, active/reactive power of loads and generators) to specific actions for each smart inverter.
  2. The Critic ($\theta$): Estimates the Q-value (expected long-term reward) of the state-action pairs, providing feedback to the Actor to improve its policy.

The core innovation is the multi-objective reward function, which incorporates three weighted terms: voltage regulation ($r_v$), action magnitude ($r_a$), and fairness ($r_f$). The fairness term uses the Gini index—a statistical measure of inequality—to punish the agent if power curtailment is distributed unevenly among customers.

Experimental Setup

The algorithm was validated using the SimBench dataset on a low-voltage network featuring 20 customers with 100% PV penetration. The agent was trained on four months of 15-minute resolution data and tested on the remaining eight months to evaluate its performance across varying seasonal weather patterns.

Key Results and Discussion

The researchers compared several control strategies, most notably Scenario C (efficiency-focused, minimizing total curtailment) and Scenario D (fairness-focused, incorporating the Gini index).

Comparison of Gini Index distributions across different control scenarios, demonstrating the reduction in inequality in Scenario d.

The results highlight a clear "Efficiency-Fairness Trade-off." While Scenario C was highly efficient—curtailing only 2.65 MWh (5.6% of total potential)—it resulted in a high Gini index, indicating that a few "end-of-line" customers bore most of the loss. In contrast, Scenario D successfully lowered the Gini index by spreading the curtailment burden across more customers. This fairness came at a cost: total energy curtailed rose to 8.77 MWh (18%).

Crucially, both scenarios successfully eliminated voltage violations, proving that fairness can be integrated into grid management without compromising safety. The agent also learned to utilize reactive power as a buffer, often resolving minor voltage issues without needing to curtail active power at all.

Conclusion and Future Directions

This research demonstrates that DRL can effectively manage complex grid constraints while addressing socio-economic concerns like fairness. By incorporating the Gini index into the reward structure, the "last-house-on-the-block" penalty can be mitigated.

Future work will focus on scaling this approach to larger, more complex network topologies and exploring decentralized reinforcement learning. In a decentralized model, inverters could communicate locally to reach a fair consensus, reducing the need for a centralized controller and increasing system resilience.

Summary updated

BibTeX
@misc{vassallo2024fair,
  author = {Maurizio Vassallo and Amina Benzerga and Alireza Bahmanyar and Damien Ernst},
  title = {{Fair Reinforcement Learning Algorithm for PV Active Control in LV Distribution Networks}},
  year = {2024},
  eprint = {2409.09074},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.09074},
}
Figure