Actor-Attention-Critic for Multi-Agent Reinforcement Learning

Shariq Iqbal, Fei Sha

Introduction

Multi-agent reinforcement learning (MARL) is notoriously difficult because the environment is non-stationary: as agents learn and change their policies, the "rules of the game" effectively shift for everyone else.

Current solutions generally fall into two camps, both with major drawbacks. Independent Learners treat other agents as part of the environment, which violates the basic Markov assumptions of RL and often fails to converge. On the other hand, Centralized Training, Decentralized Execution (CTDE) algorithms (like MADDPG) use a "centralized critic" that sees everything during training. However, these usually concatenate all agents' observations and actions into a single long vector. This leads to an input space that grows quadratically with the number of agents, making it impossible to scale to large groups. Furthermore, these models treat all agents as equally important at all times, which is rarely true in the real world.

To address these limitations, the Actor-Attention-Critic (MAAC) algorithm introduces a centralized critic that uses an attention mechanism to selectively focus on relevant information from other agents. This approach significantly improves scalability and performance across cooperative, competitive, and mixed-reward environments.

The MAAC Architecture

MAAC utilizes a centralized critic enhanced by a differentiable attention mechanism.

The MAAC architecture: each agent queries others for information, which is then aggregated via attention heads to estimate the Q-value.

Key technical components include:

  • Attention-Based Critic: Instead of concatenating all data, each agent "queries" the observations and actions of other agents. The model computes a weighted sum of other agents' information based on their current relevance.
  • Multi-Head Attention: Multiple "heads" allow the critic to focus on different subgroups of agents for different reasons simultaneously.
  • Soft Actor-Critic (SAC) Integration: MAAC builds on the SAC framework, using maximum entropy RL to encourage exploration and prevent premature convergence to sub-optimal deterministic policies.
  • Generalized Counterfactual Baseline: To solve the "credit assignment" problem (figuring out which agent's action actually caused a reward), the authors implement a multi-agent baseline that marginalizes out an agent's own actions to calculate its specific contribution.

Key Insights: Why Attention Works

The core advantage of MAAC is selective listening. Agents don't need to know everything about everyone. By learning what to pay attention to, the model filters out noise and focuses on the interactions that actually drive rewards.

Because the attention mechanism aggregates information into a constant-sized vector regardless of the number of agents, the input space grows linearly rather than quadratically. This makes MAAC much more efficient for large-scale simulations. Furthermore, by sharing the attention parameters across all agents, the model learns a universal way to represent agent interactions, effectively performing "multi-task learning" for the critics.

Experimental Results

The authors evaluated MAAC using the Multi-Agent Particle Environment framework across three distinct tasks: Cooperative Treasure Collection, Rover-Tower coordination, and Cooperative Navigation.

MAAC consistently outperformed state-of-the-art baselines like MADDPG and COMA, particularly as the complexity and number of agents increased.

Mean episode rewards across training. MAAC (blue) shows faster convergence and higher final rewards compared to baselines.
  • Scalability: In the Treasure Collection task, as the number of agents increased from 4 to 12, MAAC’s performance improvement over the next best baseline (MADDPG+SAC) jumped from 17% to a staggering 208%.
  • Emergent Coordination: In the Rover-Tower task, the model learned to have Rovers attend specifically to their assigned Towers without any explicit supervision or "pairing" signal provided during training.
  • Robustness: Unlike previous methods that were often specialized for purely cooperative tasks, MAAC proved flexible enough to handle environments with individualized and even adversarial reward structures.

Limitations and Future Work

While MAAC handles dozens of agents well, scaling to hundreds or thousands of agents might still require further innovations, such as hierarchical attention or sharing policies among groups of similar agents. Additionally, while the theory supports continuous action spaces, the experimental results focused primarily on discrete actions to simplify baseline comparisons. Future work could explore more complex, high-dimensional continuous control tasks in multi-agent settings.

Summary updated

Abstract

Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings. We present an actor-critic algorithm tha...
BibTeX
@inproceedings{iqbal2019actor,
  author = {Shariq Iqbal and Fei Sha},
  title = {{Actor-Attention-Critic for Multi-Agent Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=BkZxxjW_bS},
}
Figure