Multi-Agent Actor-Critic with Generative Cooperative Policy Network

Heechang Ryu, Hayong Shin, Jinkyoo ParkarXiv 2018 · arXiv

Abstract

Multi-agent reinforcement learning (MARL) often struggles with non-stationarity and the credit assignment problem, particularly when agents are trained with individual rewards. This paper proposes MADDPG-GCPN, which introduces a Generative Cooperative Policy Network (GCPN) to each agent. The GCPN is designed to generate training samples that maximize the rewards of other agents, encouraging the discovery of cooperative joint policies. Experimental results in predator-prey scenarios and real-world microgrid energy management demonstrate that MADDPG-GCPN achieves higher cooperation and better system-wide performance compared to standard MADDPG.

Introduction

In complex multi-agent systems, such as autonomous vehicle fleets or smart power grids, agents must coordinate their actions to achieve global objectives. However, training these agents is challenging because the environment is non-stationary—as one agent learns, the environment perceived by others changes. While shared global rewards can mitigate selfish behavior, they introduce the "credit assignment" problem, where it is unclear how an individual's action contributed to the team's success. Conversely, individual rewards often lead to selfish, sub-optimal policies. MADDPG-GCPN addresses this by enabling decentralized agents with individual rewards to learn cooperative strategies through "altruistic" exploration during training.

Methods

The proposed MADDPG-GCPN algorithm extends the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) framework, which follows the "centralized training, decentralized execution" paradigm.

Dual-Actor Architecture

The core innovation is a dual-actor structure for each agent. While standard MADDPG uses one actor and one critic, MADDPG-GCPN employs:

  1. Individual Greedy Policy Network: This network is trained to maximize the agent's own return and is used for decentralized execution.
  2. Generative Cooperative Policy Network (GCPN): Used only during the centralized training phase, this network is optimized using the gradients of other agents' critics. Its purpose is to explore actions that benefit the agent's peers.
Comparison of standard MADDPG and the proposed MADDPG-GCPN architecture.

During training, the GCPN generates exploratory samples for the replay buffer. These samples are biased toward cooperative outcomes, forcing the system to explore regions of the policy space that benefit the collective. During execution, the GCPNs are discarded, maintaining decentralized control and privacy.

Experiments and Results

The authors evaluated MADDPG-GCPN in two distinct environments: a simulated predator-prey game and a real-world microgrid energy storage system (ESS) control task.

Predator-Prey Simulation

In this environment, three slower predators must cooperate to capture a faster prey while navigating obstacles. In the difficult individual-reward setting, MADDPG-GCPN significantly outperformed baselines like standard MADDPG and FDMARL in terms of prey-catching rate and normalized scores.

The predator-prey environment where three predators (red) must cooperate to catch a faster prey (green).

Microgrid Energy Management

Using historical demand and wind energy data, the algorithm was applied to decentralized microgrid control. Standard independent DDPG agents exhibited selfish behavior, leading to high energy costs. MADDPG-GCPN achieved total costs near the theoretical optimal (calculated via centralized Quadratic Programming) and effectively reduced the Peak-to-Average Ratio (PAR), enhancing grid stability.

Conclusion

MADDPG-GCPN provides a robust framework for fostering cooperation in MARL without requiring shared rewards. By utilizing a secondary generative network for altruistic exploration, agents can discover mutually beneficial strategies while maintaining decentralized execution. Future work will focus on reducing the computational overhead associated with training the additional GCPN networks.

Summary updated

Abstract

We propose an efficient multi-agent reinforcement learning approach to derive equilibrium strategies for multi-agents who are participating in a Markov game. Mainly, we are focused on obtaining decentralized policies for agents to maximize the performance of a collaborative task by all the agents, which is similar to solving a decentralized Markov decision process. We propose to use two different policy networks: (1) decentralized greedy policy network used to generate greedy action during training and execution period and (2) generative cooperative policy network (GCPN) used to generate action samples to make other agents improve their objectives during training period. We show that the samples generated by GCPN enable other agents to explore the policy space more effectively and favorably to reach a better policy in terms of achieving the collaborative tasks.
BibTeX
@misc{ryu2018multi,
  author = {Heechang Ryu and Hayong Shin and Jinkyoo Park},
  title = {{Multi-Agent Actor-Critic with Generative Cooperative Policy Network}},
  year = {2018},
  eprint = {1810.09206},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.09206},
}
Figure