Scalable Communication for Multi-Agent Reinforcement Learning via Transformer-Based Email Mechanism

Xudong Guo, Daming Shi, Wenhui FanIJCAI 2023 · DOI

TL;DR

This paper introduces the Transformer-based Email Mechanism (TEM), a novel communication framework for multi-agent reinforcement learning (MARL) that addresses the scalability challenge. TEM enables agents to selectively send and forward messages, forming "message chains" akin to email forwarding, allowing for efficient information sharing even when agents cannot directly observe each other. This approach significantly improves performance in cooperative MARL tasks and demonstrates remarkable scalability to varying numbers of agents without retraining.

What Problem Does This Paper Address?

Multi-agent reinforcement learning (MARL) has shown promise in complex tasks, but achieving effective cooperation, especially in partially-observed environments, remains a significant challenge. Existing communication methods in MARL often suffer from scalability issues. Some methods broadcast messages to all agents, leading to information redundancy and high communication overhead. Others attempt targeted peer-to-peer communication but require agents to model every other agent, which becomes computationally prohibitive as the number of agents increases. Furthermore, these methods struggle to adapt when the number of agents changes, necessitating retraining. This paper tackles the problem of developing a communication mechanism that is both effective in partially-observed MARL and scalable to a varying number of agents.

Methods

The core contribution of this paper is the Transformer-based Email Mechanism (TEM). TEM introduces a novel communication protocol inspired by human email forwarding. Key aspects include:

  • Local Communication: Agents only send messages to other agents within their observation range.
  • Message Chains: To share information beyond direct observation, agents can forward messages. This creates "message chains" where information is passed along, similar to how emails are forwarded in a chain.
  • Transformer-based Encoding/Decoding: A Transformer network is used to encode and decode these message chains. This allows agents to selectively choose the next receiver for a message, ensuring that information is passed efficiently and only to agents who can benefit from it.
  • Selective Communication: Agents learn to decide whether to communicate and whom to communicate with, based on their observations and the messages they have received. This is achieved through a learned communication action that can either be "do not send" or "send to agent j" (where j is within the observation range).
  • Message Buffer: Each agent maintains a message buffer (implemented as a queue) to store received messages, allowing for the processing of message chains.
  • Independent Communication Module: The TEM communication module is designed to be pluggable into existing MARL frameworks, specifically using MAPPO (Multi-Agent Proximal Policy Optimization) as a backbone.
  • Novel Loss Function: A specific loss function is designed to encourage agents to send messages that have a positive impact on other agents' actions, while also penalizing unnecessary communication.

Datasets

The paper evaluates TEM on three widely-used partially-observed cooperative MARL benchmarks:

  • Starcraft Multi-Agent Challenge (SMAC): Specifically, the "5m vs. 6m" map.
  • Predator Prey (PP): A 7-3 scenario (7 agents, 3 preys).
  • Cooperative Navigation (CN): A 7-7 scenario (7 agents, 7 landmarks).

The paper also tests the scalability of TEM by transferring models trained on these scenarios to new environments with different numbers of agents and targets without further training.

Key Insights and Ideas

  • Email Forwarding as a Communication Paradigm: The analogy to email forwarding is a powerful and intuitive way to conceptualize scalable, indirect communication in MARL. It highlights how information can propagate through a network even without direct connections between all nodes.
  • Scalability through Indirectness: By avoiding the need to model every agent and instead relying on local communication and message forwarding, TEM achieves scalability. This is a crucial step towards applying MARL to real-world problems with potentially hundreds or thousands of agents.
  • Selective Communication is Key: The ability to selectively send and forward messages, rather than broadcasting, significantly reduces communication overhead and improves efficiency. The Transformer's role in selecting the next receiver is central to this.
  • Decentralization and Practicality: The proposed mechanism is decentralized, making it more practical for real-world applications where a central controller might not be feasible.
  • Zero-Shot Transferability: The remarkable ability of TEM to transfer to new scenarios with different numbers of agents without retraining is a strong indicator of its robust and generalizable communication strategy.

Key Results

  • Superior Performance: TEM consistently outperforms baseline methods (MAPPO, MADDPG, etc.) on SMAC, Predator Prey, and Cooperative Navigation tasks. For instance, in SMAC, TEM achieves the highest win rate. In Predator Prey, it improves the final reward by 17.2% compared to MAPPO.
  • Scalability Demonstrated: When transferred to scenarios with more or fewer agents (e.g., 9-3 and 3-1 for Predator Prey, 6-6 and 9-9 for Cooperative Navigation) without retraining, TEM maintains superior performance. In some cases, transferred TEM even outperforms MAPPO trained from scratch on the new scenarios.
  • Reduced Internal Strife: In Cooperative Navigation, illustrations show that TEM's communication helps agents avoid collisions and internal conflicts by guiding them to different landmarks, reducing "internal strife."

Limitations and Future Work

  • Observation Range: The current implementation of TEM considers agents within the "nearest several agents" as the observation range for communication. The paper suggests that exploring different definitions of this range could be a future direction.
  • Message Content: While TEM focuses on the mechanism of communication, future work could explore more sophisticated ways to generate and interpret message content.
  • Complex Scenarios: The paper mentions that for very simple environments, the communication aspect might not be as critical. Future research could investigate the trade-offs between communication complexity and task simplicity.
  • Broader Applications: The authors suggest that TEM's scalability makes it a promising step for MARL applications involving hundreds of agents and for developing more tailored message generation strategies for real-world tasks.

Summary updated

BibTeX
@inproceedings{guo2023scalable,
  author = {Xudong Guo and Daming Shi and Wenhui Fan},
  title = {{Scalable Communication for Multi-Agent Reinforcement Learning via Transformer-Based Email Mechanism}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/15},
}
Figure