Actor-Attention-Critic for Multi-Agent Reinforcement Learning

Shariq Iqbal, Fei Sha

Abstract

Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings. We present an actor-critic algorithm that trains decentralized policies in multi-agent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep. This attention mechanism enables more effective and scalable learning in complex multi-agent environments, when compared to recent approaches. Our approach is applicable not only to cooperative settings with shared rewards, but also individualized reward settings, including adversarial settings, as well as settings that do not provide global states, and it makes no assumptions about the action spaces of the agents. As such, it is flexible enough to be applied to most multi-agent learning problems.

Introduction

Training multiple agents simultaneously is inherently difficult. In independent training, the environment becomes non-stationary from any single agent's perspective because other agents are constantly updating their strategies. To mitigate this, researchers often employ Centralized Training with Decentralized Execution (CTDE). In this paradigm, a "critic" has access to global information during training to provide stable feedback, while "actors" (the agents) rely only on local observations during execution.

However, existing CTDE methods like MADDPG face scalability issues. Their critics typically concatenate the observations and actions of all agents into a single large vector. As the number of agents grows, this input space expands quadratically, making it difficult for the critic to discern which information is relevant. This "brute-force" approach is inefficient for large-scale coordination.

Methodology: Multi-Actor-Attention-Critic (MAAC)

The core innovation of MAAC is the replacement of the standard concatenated critic with an Attention-based Critic. This mechanism allows each agent's critic to selectively process information from its peers.

MAAC architecture diagram showing the attention mechanism within the centralized critic.

The architecture consists of several key components:

  1. Attention Mechanism: Inspired by Transformer models, each agent’s critic "queries" other agents for information. Using a query-key-value system, it calculates a weighted sum of contributions. Agents that are more relevant to the current state receive higher attention weights, effectively filtering out background noise.
  2. Soft Actor-Critic (SAC) Integration: MAAC utilizes SAC as a backbone, incorporating an entropy term to encourage exploration and prevent premature convergence to sub-optimal strategies.
  3. Shared Feature Learning: Critics share parameters, enabling multi-task learning. This allows the model to learn common environmental features even when agents have distinct reward functions.
  4. Counterfactual Baseline: To address the multi-agent credit assignment problem, the authors implement a baseline that marginalizes out an agent's own action, helping to quantify its specific contribution to the total reward.

Experimental Setup

The performance of MAAC was evaluated using the Multi-Agent Particle Environment (MPE) across three tasks:

  • Cooperative Treasure Collection: 6 hunters collect treasure for 2 banks, testing scalability and mixed reward structures.
  • Rover-Tower: 4 rovers must reach hidden goals by communicating with 4 towers, testing the ability to form specific communication pairings.
  • Cooperative Navigation: A benchmark task requiring agents to cover landmarks while avoiding collisions.

Key Results and Discussion

MAAC demonstrated significant improvements in scalability and coordination compared to traditional baselines.

Comparison of MAAC against various baselines across different multi-agent environments.
  • Scalability: In the Treasure Collection task, MAAC's performance remained stable as agent counts increased, whereas MADDPG's performance degraded. With 12 agents, MAAC achieved a 208% improvement over the next best baseline.
  • Dynamic Coordination: In the Rover-Tower task, the attention mechanism successfully learned to focus on specific paired partners without explicit supervision.
  • Generalization: MAAC proved robust across cooperative, competitive, and mixed-reward settings, establishing it as a versatile framework for MARL.

Conclusion

MAAC addresses the scalability bottleneck of centralized critics by employing selective attention. While the current framework focuses on the critic, future work could explore information sharing between decentralized actors or hierarchical attention structures for even larger populations. The results indicate that selective attention is a powerful tool for managing the complexity of multi-agent interactions.

Summary updated

Abstract

Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings. We present an actor-critic algorithm that trains decentralized policies in multi-agent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep. This attention mechanism enables more effective and scalable learning in complex multi-agent environments, when compared to recent approaches. Our approach is applicable not only to cooperative settings with shared rewards, but also individualized reward settings, including adversarial settings, as well as settings that do not provide global states, and it makes no assumptions about the action spaces of the agents. As such, it is flexible enough to be applied to most multi-agent learning problems.
BibTeX
@inproceedings{iqbal2019actor,
  author = {Shariq Iqbal and Fei Sha},
  title = {{Actor-Attention-Critic for Multi-Agent Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=BkZxxjW_bS},
}
Figure