Expressive Multi-Agent Communication via Identity-Aware Learning

Wei Du, Shifei Ding, Lili Guo, Jian Zhang, Ling Ding

Introduction: The Diversity Bottleneck in MARL

In Multi-Agent Reinforcement Learning (MARL), effective communication is essential for coordination. However, standard communication protocols often rely on Graph Neural Networks (GNNs) that treat agents as interchangeable nodes. This leads to a "diversity bottleneck" where agents with similar local neighborhoods develop nearly identical feature representations. Theoretically, these methods are upper-bounded by the 1-Weisfeiler-Lehman (1-WL) test, making it difficult for agents to break symmetry and adopt specialized roles—such as one defender challenging a ball while another covers.

The IDEAL Framework

To address this, we introduce IDEAL (Identity-Aware Learning), a module designed to enhance the expressivity of GNN-based communication. IDEAL breaks the 1-WL bottleneck through two primary mechanisms:

  1. Inductive Identity Coloring: Each "ego" agent assigns a unique tag to itself during message processing. This differentiates the self from neighbors within the local computational graph without relying on fixed global IDs.
  2. Heterogeneous Message Passing: IDEAL employs distinct parameter sets for the "colored" ego agent and its "uncolored" neighbors. By processing self-information differently from neighbor-information, the network produces more distinguishable features.
Comparison of computational graphs between standard GNNs and IDEAL, showing how identity coloring breaks symmetry.

Experimental Setup

The IDEAL module was evaluated across four diverse MARL benchmarks, ranging from simple grid worlds to complex strategic simulations:

  • Predator-Prey: Coordination to capture prey in a grid.
  • Traffic Junction: Collision-free navigation at busy intersections.
  • Battle (MAgent): Large-scale combat requiring complex formations.
  • SMAC (StarCraft Multi-Agent Challenge): High-fidelity micro-management tasks.
The four MARL benchmarks used to evaluate IDEAL: Predator-Prey, Traffic Junction, Battle, and SMAC.

Key Results and Insights

IDEAL consistently outperformed eight state-of-the-art baselines (e.g., TarMAC, MAGIC, DICG) across all environments:

  • Improved Coordination: In Predator-Prey (20x20), IDEAL reduced capture time from 36.22 to 31.57 steps.
  • Higher Success Rates: In the "Hard" Traffic Junction scenario, it achieved an 82.45% success rate compared to the baseline's 73.26%.
  • Strategic Depth: In SMAC's "MMM2" scenario, IDEAL reached an 89.23% win rate, significantly higher than the standard DICG's 83.45%.

A key finding is that the performance gap between IDEAL and traditional GNNs widens as the number of agents increases. This suggests that identity-aware learning is particularly effective at managing the increased neighborhood similarity found in crowded environments. Furthermore, IDEAL achieves this expressivity with negligible overhead in training time.

Conclusion

IDEAL demonstrates that explicitly incorporating agent identity into communication protocols is vital for breaking behavioral homogeneity. By allowing agents to recognize their unique positions within a team, IDEAL enables the emergence of complex, coordinated strategies necessary for solving sophisticated multi-agent tasks. Future work will explore these techniques in bandwidth-constrained and noisy real-world robotic environments.

Summary updated

Abstract

Information sharing through communication is essential for tackling complex multi-agent reinforcement learning tasks. Many existing multi-agent communication protocols can be viewed as instances of message passing graph neural networks (GNNs). However, due to the significantly limited expressive ability of the standard GNN method, the agent feature representations remain similar and indistinguishable even though the agents have different neighborhood structures. This further results in the homogenization of agent behaviors and reduces the capability to solve tasks effectively. In this paper, we propose a multi-agent communication protocol via identity-aware learning (IDEAL), which explicitly enhances the distinguishability of agent feature representations to break the diversity bottleneck. Specifically, IDEAL extends existing multi-agent communication protocols by inductively considering the agents' identities during the message passing process. To obtain expressive feature representations for a given agent, IDEAL first extracts the ego network centered around that agent and then performs multiple rounds of heterogeneous message passing, where different parameter sets are applied to the central agent and the other surrounding agents within the ego network. IDEAL fosters expressive communication between agents and generates distinguishable feature representations, which promotes action diversity and individuality emergence. Experimental results on various benchmarks demonstrate IDEAL can be flexibly integrated into various multi-agent communication methods and enhances the corresponding performance.
BibTeX
@inproceedings{du2024expressive,
  author = {Wei Du and Shifei Ding and Lili Guo and Jian Zhang and Ling Ding},
  title = {{Expressive Multi-Agent Communication via Identity-Aware Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29683},
}
Figure