MARL for Decentralized Electric Vehicle Charging Coordination with V2V Energy Exchange

Jiarong Fan, Hao Wang, Ariel Liebman

Abstract

Effective energy management of electric vehicle (EV) charging stations is critical to supporting the transport sector's sustainable energy transition. This paper addresses the EV charging coordination by considering vehicle-to-vehicle (V2V) energy exchange as the flexibility to harness in EV charging stations. Moreover, this paper takes into account EV user experiences, such as charging satisfaction and fairness. We propose a Multi-Agent Reinforcement Learning (MARL) approach to coordinate EV charging with V2V energy exchange while considering uncertainties in the EV arrival time, energy price, and solar energy generation. The exploration capability of MARL is enhanced by introducing parameter noise into MARL's neural network models. Experimental results demonstrate the superior performance and scalability of our proposed method compared to traditional optimization baselines. The decentralized execution of the algorithm enables it to effectively deal with partial system faults in the charging station.

Introduction

As the global transport sector transitions toward sustainable energy, the management of electric vehicle (EV) charging stations has become increasingly complex. Station operators must navigate a landscape of volatile electricity prices, intermittent renewable energy sources like solar power, and the unpredictable behavior of EV drivers. Traditional management systems often struggle to balance operational costs with user satisfaction, particularly when faced with real-world uncertainties.

This research introduces a decentralized Multi-Agent Reinforcement Learning (MARL) framework designed to coordinate EV charging while integrating vehicle-to-vehicle (V2V) energy exchange. By leveraging V2V flexibility and prioritizing user fairness, the proposed system offers a resilient and scalable solution for modern energy infrastructure.

The Challenge of Coordination

Existing EV charging coordination strategies generally fall into two categories, both of which possess significant drawbacks:

  1. Model-based approaches: These systems rely on precise future predictions. If electricity prices spike unexpectedly or weather patterns shift, the performance of these models degrades rapidly.
  2. Centralized Model-free approaches: While these handle uncertainty better, they require constant, high-bandwidth communication between all chargers and a central controller. This creates a single point of failure; if the central link or controller fails—a "partial system fault"—the entire coordination effort can collapse.

Furthermore, many algorithms focus exclusively on cost minimization, neglecting user satisfaction and fairness. A system that leaves one vehicle empty to save costs while fully charging another is unlikely to achieve widespread user adoption.

Proposed Methodology

The study proposes a decentralized MARL approach based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. This framework is characterized by several key technical innovations:

Centralized Training, Decentralized Execution (CTDE)

The agents (individual chargers) undergo a training phase where they have access to global information, allowing them to learn optimal collaborative strategies. However, during deployment, each charger operates independently, making decisions based solely on its local state, such as the battery level of the connected EV. This ensures the system remains operational even if communication with a central node is lost.

V2V Energy Exchange and Solar Integration

The model incorporates V2V energy exchange, allowing EVs to trade energy among themselves. This creates a local energy buffer that reduces the station's reliance on the grid during peak price periods. This is supplemented by a 30kWp local photovoltaic (PV) system to further offset costs and promote sustainability.

Enhanced Exploration via Noisy Networks

To improve the learning process, the researchers implemented "parameter noise" within the neural network weights (NoisyNet). Unlike standard methods that add noise to the agents' actions, perturbing the network parameters encourages more diverse and efficient exploration of the strategy space, leading to faster and more stable convergence.

Fairness-Aware Reward Function

The reward function is designed to balance economic efficiency with social equity. It utilizes Future Average Power (FAP) to measure charging satisfaction and includes a fairness factor that penalizes the system if the satisfaction gap between different users becomes too wide.

Experimental Setup and Data

The framework was validated using a combination of real-world datasets:

  • EV Charging Behavior: Data from the Adaptive Charging Network (ACN) provided realistic arrival/departure times and energy demands.
  • Energy Markets: Real-time and Time-of-Use (TOU) pricing data were sourced from the National Electricity Market (NEM) in Australia.
  • Renewables: Solar generation data was synchronized with the charging and pricing timelines to simulate a realistic local microgrid.

Key Insights and Results

Resilience Through Decentralization

A critical finding of the study was the system's resilience to partial faults. When random noise was injected into some chargers to simulate hardware or communication failures, centralized baselines (such as MADQN) became unstable. In contrast, the decentralized MADDPG approach continued to function normally, as agents did not rely on potentially faulty information from their neighbors.

The Cost of Fairness

The research quantified the "fairness trade-off." While prioritizing user equity slightly increases the aggregate energy cost, it significantly reduces the disparity in charge levels between users. This trade-off is essential for maintaining user trust and long-term system viability.

Scalability and Performance

The algorithm demonstrated excellent scalability, maintaining a consistent average energy cost per charger as the station size increased from 10 to 80 chargers. In comparative tests, the MARL approach consistently outperformed traditional Rolling-Horizon Optimization (RHO), achieving an average charging completion ratio of over 80% while maintaining lower costs than the baselines.

Conclusion and Future Work

This decentralized MARL framework provides a robust solution for EV charging coordination, successfully balancing cost, resilience, and user fairness. While the current model uses a simplified V2V pricing structure, future research will focus on integrating complex grid constraints, such as local transformer limits and voltage fluctuations. Ensuring that decentralized charging supports the stability of the wider power grid remains the next frontier for this technology.

Summary updated

Abstract

Effective energy management of electric vehicle (EV) charging stations is critical to supporting the transport sector's sustainable energy transition. This paper addresses the EV charging coordination by considering vehicle-to-vehicle (V2V) energy exchange as the flexibility to harness in EV charging stations. Moreover, this paper takes into account EV user experiences, such as charging satisfaction and fairness. We propose a Multi-Agent Reinforcement Learning (MARL) approach to coordinate EV charging with V2V energy exchange while considering uncertainties in the EV arrival time, energy price, and solar energy generation. The exploration capability of MARL is enhanced by introducing parameter noise into MARL's neural network models. Experimental results demonstrate the superior performance and scalability of our proposed method compared to traditional optimization baselines. The decentralized execution of the algorithm enables it to effectively deal with partial system faults in the charging station.
BibTeX
@inproceedings{fan2023marl,
  author = {Jiarong Fan and Hao Wang and Ariel Liebman},
  title = {{MARL for Decentralized Electric Vehicle Charging Coordination with V2V Energy Exchange}},
  booktitle = {IECON},
  year = {2023},
  url = {https://openreview.net/forum?id=6cQqEm67T8},
}
Figure