Abstract
Multi-agent reinforcement learning (MARL) often struggles with non-stationarity and the credit assignment problem, particularly when agents are trained with individual rewards. This paper proposes MADDPG-GCPN, which introduces a Generative Cooperative Policy Network (GCPN) to each agent. The GCPN is designed to generate training samples that maximize the rewards of other agents, encouraging the discovery of cooperative joint policies. Experimental results in predator-prey scenarios and real-world microgrid energy management demonstrate that MADDPG-GCPN achieves higher cooperation and better system-wide performance compared to standard MADDPG.
Introduction
In complex multi-agent systems, such as autonomous vehicle fleets or smart power grids, agents must coordinate their actions to achieve global objectives. However, training these agents is challenging because the environment is non-stationary—as one agent learns, the environment perceived by others changes. While shared global rewards can mitigate selfish behavior, they introduce the "credit assignment" problem, where it is unclear how an individual's action contributed to the team's success. Conversely, individual rewards often lead to selfish, sub-optimal policies. MADDPG-GCPN addresses this by enabling decentralized agents with individual rewards to learn cooperative strategies through "altruistic" exploration during training.
Methods
The proposed MADDPG-GCPN algorithm extends the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) framework, which follows the "centralized training, decentralized execution" paradigm.
Dual-Actor Architecture
The core innovation is a dual-actor structure for each agent. While standard MADDPG uses one actor and one critic, MADDPG-GCPN employs:
- Individual Greedy Policy Network: This network is trained to maximize the agent's own return and is used for decentralized execution.
- Generative Cooperative Policy Network (GCPN): Used only during the centralized training phase, this network is optimized using the gradients of other agents' critics. Its purpose is to explore actions that benefit the agent's peers.
During training, the GCPN generates exploratory samples for the replay buffer. These samples are biased toward cooperative outcomes, forcing the system to explore regions of the policy space that benefit the collective. During execution, the GCPNs are discarded, maintaining decentralized control and privacy.
Experiments and Results
The authors evaluated MADDPG-GCPN in two distinct environments: a simulated predator-prey game and a real-world microgrid energy storage system (ESS) control task.
Predator-Prey Simulation
In this environment, three slower predators must cooperate to capture a faster prey while navigating obstacles. In the difficult individual-reward setting, MADDPG-GCPN significantly outperformed baselines like standard MADDPG and FDMARL in terms of prey-catching rate and normalized scores.
Microgrid Energy Management
Using historical demand and wind energy data, the algorithm was applied to decentralized microgrid control. Standard independent DDPG agents exhibited selfish behavior, leading to high energy costs. MADDPG-GCPN achieved total costs near the theoretical optimal (calculated via centralized Quadratic Programming) and effectively reduced the Peak-to-Average Ratio (PAR), enhancing grid stability.
Conclusion
MADDPG-GCPN provides a robust framework for fostering cooperation in MARL without requiring shared rewards. By utilizing a secondary generative network for altruistic exploration, agents can discover mutually beneficial strategies while maintaining decentralized execution. Future work will focus on reducing the computational overhead associated with training the additional GCPN networks.
Summary updated