A Review of Cooperative Multi-Agent Deep Reinforcement Learning

Afshin Oroojlooyjadid, Davood HajinezhadarXiv 2019 · OpenReview

TL;DR

This paper provides a comprehensive review of cooperative Multi-Agent Deep Reinforcement Learning (MARL) algorithms, categorizing recent approaches into five key areas: independent learners, fully observable critics, value function factorization, consensus, and learning to communicate. It aims to synthesize the rapidly evolving field, highlight challenges and solutions, and identify promising future research directions and applications.

What Problem Does This Paper Address?

Multi-agent systems, where multiple agents interact within a shared environment to achieve individual or collective goals, are becoming increasingly prevalent in real-world applications. However, training these systems using Reinforcement Learning (RL) presents significant challenges. Unlike single-agent RL, MARL faces issues like non-stationarity (as other agents' policies change, the environment appears to change from an individual agent's perspective), the curse of dimensionality in joint action spaces, and the need for effective coordination and communication. This paper specifically focuses on cooperative MARL, where agents work together towards a common objective, and aims to provide a structured overview of the diverse approaches developed to tackle these challenges.

Methods

The paper categorizes and reviews MARL algorithms based on five core approaches:

  • Independent Learners (IQL): Treats each agent as an independent learner, considering other agents as part of the environment. This is a simple extension of single-agent RL but suffers from non-stationarity.
  • Fully Observable Critic: Addresses non-stationarity by having a critic observe the global state and actions of all agents. This allows for a stationary environment from the critic's perspective.
  • Value Function Factorization: Decomposes the joint value function into individual agent value functions to handle large action spaces and avoid issues like "lazy agents" by explicitly attributing contributions to the global reward.
  • Consensus: Focuses on enabling agents to communicate through a sparse network to reach a shared understanding or policy, aiming to avoid overwhelming a central unit.
  • Learn to Communicate: Goes beyond simply sharing information by allowing agents to learn what to send, when to send, and to whom to send messages, treating communication as an action itself.

The paper also discusses extensions and variations within these categories, including actor-critic methods, deep learning architectures (CNNs, RNNs), and techniques for handling partial observability (POMDPs).

Datasets

Not explicitly detailed as datasets in the paper. However, the paper references numerous environments used for evaluating MARL algorithms, including:

  • Atari games: For single-agent RL benchmarks, often extended to multi-agent settings.
  • StarCraft II: A complex real-time strategy game frequently used for MARL research.
  • Multi-Agent Particle Environment: A suite of tasks including predator-prey, navigation, and arrival tasks.
  • Traffic signal control environments: Such as CityFlow.
  • Robotics simulation environments: For path planning and coordination tasks.
  • Games: Like "Guess Who?", Pong, and others for communication and coordination studies.

Key Insights and Ideas

  • Non-stationarity is a fundamental challenge: The changing policies of other agents make the environment appear non-stationary to individual agents, a core problem addressed by many MARL techniques.
  • Centralized training with decentralized execution (CTDE) is a dominant paradigm: Many advanced MARL algorithms leverage CTDE, where a centralized critic can access global information during training, but agents act independently during execution.
  • Communication is not just about sharing data, but learning how to communicate: The "Learn to Communicate" category highlights the shift towards agents learning to strategically send and interpret messages, enabling more efficient coordination.
  • Value function factorization is crucial for scalability: Decomposing joint value functions is essential for handling the combinatorial explosion of joint actions in multi-agent systems.
  • Real-world applications are diverse and driving MARL research: From traffic control and robotics to resource allocation and stock markets, the demand for intelligent multi-agent coordination is fueling innovation.

Key Results

The paper is a review and does not present novel experimental results. Instead, it synthesizes findings from numerous research papers. The key "results" are the identification of successful strategies and algorithms that have demonstrated effectiveness in various MARL settings, including:

  • Improved coordination: Algorithms like those using value function factorization and consensus mechanisms have shown success in cooperative tasks.
  • Efficient communication: Learn-to-communicate approaches have enabled agents to achieve better performance by strategically exchanging information.
  • Scalability: Techniques like value function factorization and attention mechanisms are crucial for scaling MARL to systems with many agents.
  • Application success: MARL has been successfully applied to complex real-world problems like traffic control, robot path planning, and resource management.

Limitations and Future Work

Limitations:

  • Scalability: While progress has been made, scaling MARL to very large numbers of agents remains a significant challenge.
  • Partial Observability: Many real-world scenarios involve agents with limited information, and effectively handling this (Dec-POMDPs) is an ongoing area of research.
  • Heterogeneous Agents: Most current MARL research assumes homogeneous agents; developing algorithms for agents with different skills and abilities is a key limitation.
  • Theoretical Guarantees: Despite impressive empirical performance, theoretical convergence guarantees and formal analysis for deep MARL algorithms are still lacking.

Future Work:

  • Off-policy MARL: Further research is needed to explore and develop more effective off-policy MARL algorithms.
  • Safe MARL: Ensuring safety constraints and avoiding catastrophic situations during training and execution is a critical area for future work.
  • Heterogeneous MARL: Developing algorithms that can effectively combine the diverse abilities of heterogeneous agents is a promising direction.
  • Optimization in MARL: Advancing optimization techniques specifically for MARL problems is crucial for improving efficiency.
  • Inverse MARL: Exploring how to infer reward functions from agent behavior could unlock new applications.
  • Model-based MARL: Investigating model-based approaches could improve sample efficiency, a common limitation of model-free methods.

Summary updated

Abstract

Deep Reinforcement Learning has made significant progress in multi-agent systems in recent years. In this review article, we have focused on presenting recent approaches on Multi-Agent Reinforcement Learning (MARL) algorithms. In particular, we have focused on five common approaches on modeling and solving cooperative multi-agent reinforcement learning problems: (I) independent learners, (II) fully observable critic, (III) value function factorization, (IV) consensus, and (IV) learn to communicate. First, we elaborate on each of these methods, possible challenges, and how these challenges were mitigated in the relevant papers. If applicable, we further make a connection among different papers in each category. Next, we cover some new emerging research areas in MARL along with the relevant recent papers. Due to the recent success of MARL in real-world applications, we assign a section to provide a review of these applications and corresponding articles. Also, a list of available environments for MARL research is provided in this survey. Finally, the paper is concluded with proposals on the possible research directions.
BibTeX
@misc{oroojlooyjadid2019review,
  author = {Afshin Oroojlooyjadid and Davood Hajinezhad},
  title = {{A Review of Cooperative Multi-Agent Deep Reinforcement Learning}},
  year = {2019},
  eprint = {1908.03963},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.03963},
}
Figure