Heterogeneous Multi-Agent Communication Learning via Graph Information Maximization

Wei Du, Shifei DingSEKE 2023 · OpenReview

Abstract

In Multi-Agent Reinforcement Learning (MARL), coordinating heterogeneous agents with diverse observation and action spaces remains a significant challenge. Traditional communication methods often rely on homogeneous Graph Neural Networks (GNNs) that fail to account for distinct agent roles, leading to noisy and ineffective information exchange. This paper presents CLAR (Communication Learning of multi-Agent Reinforcement learning), a framework that leverages heterogeneous graph modeling and Mutual Information (MI) maximization to facilitate effective communication among diverse agents. Experimental results on the Predator-Capture-Prey and StarCraft II Multi-Agent Challenge (SMAC) benchmarks demonstrate that CLAR significantly outperforms state-of-the-art baselines in complex, heterogeneous scenarios.

Introduction

Most existing MARL research operates under the assumption of homogeneity, where agents share identical capabilities and perspectives. However, real-world applications—ranging from heterogeneous robot swarms to complex tactical simulations—require coordination between agents with vastly different sensors and actuators. When standard GNN-based communication is applied to these systems, the indiscriminate aggregation of features often results in "communication noise," where critical role-specific information is lost.

To address this, the researchers from China University of Mining and Technology developed CLAR. The framework focuses on two primary objectives: making communication type-aware through heterogeneous graph attention and ensuring message relevance through information theory.

Methodology

The CLAR architecture is structured into three distinct layers: feature encoding, communication, and value decomposition.

The CLAR architecture featuring feature encoding, heterogeneous communication with MI maximization, and value decomposition.

1. Feature Encoding

Each agent $i$ processes its local observation $o_i$ through an MLP and a Gated Recurrent Unit (GRU) to generate a hidden state $h_i$. This state captures the temporal dependencies of the agent's experience. To support heterogeneous communication, an additional encoder transforms these hidden states into class-specific feature representations.

2. Heterogeneous Communication via HGNN

Instead of treating all neighbors equally, CLAR utilizes a Heterogeneous Graph Attention Network (HGAT). This module employs type-specific weight matrices for different "edge types" (e.g., a message from a 'scout' to an 'attacker'). This allows the network to learn unique relational dynamics between different roles, ensuring that agents can filter and prioritize information based on the sender's class.

3. Mutual Information (MI) Maximization

A core innovation of CLAR is the use of MI maximization to refine communication. To prevent the transmission of irrelevant data, the authors introduce an MI loss function. By maximizing the correlation between received messages $m_i$ and the agent's local action-value $Q_i$, the system forces the communication module to produce "expressive" messages that are directly beneficial for decision-making. This is implemented using a discriminator that distinguishes between true message-value pairs and shuffled ones.

4. Value Factorization

CLAR follows the Centralized Training, Decentralized Execution (CTDE) paradigm. It integrates a mixing network (similar to QMIX or QPLEX) that decomposes the joint team reward into individual agent values, ensuring stable training while allowing agents to act independently during execution based on their local observations and received messages.

Experimental Evaluation

The framework was evaluated on two challenging benchmarks:

  • Predator-Capture-Prey (PCP): A task requiring coordination between "predators" (who observe the environment) and "captures" (who are blind but must catch the prey based on predator instructions).
  • StarCraft II Multi-Agent Challenge (SMAC): Tests were conducted on heterogeneous maps such as 2c3s5z (2 Colossi, 3 Stalkers, 5 Zealots) and MMM2.

Key Results

CLAR demonstrated substantial performance gains across all tested environments. In the highly complex 2c3s5z map of SMAC, CLAR achieved a peak win rate of approximately 93%, whereas standard QMIX reached only ~78% and the communication-based TarMAC reached ~60%.

Win rate comparison on the SMAC 2c3s5z map, demonstrating CLAR's superior performance over baselines.

Ablation studies confirmed the necessity of both the heterogeneous modeling and the MI optimization. Removing the MI loss (CLAR-M) or the heterogeneous GNN components (CLAR-H) resulted in significant performance degradation, proving that role-aware filtering and message relevance are both critical for heterogeneous coordination.

Furthermore, visualization of the message space using t-SNE revealed that MI optimization leads to distinct clustering of agent communications based on their functional roles. This indicates that the agents successfully learned a specialized "language" tailored to their specific tasks within the team.

Conclusion

CLAR provides a robust solution for the communication challenges inherent in heterogeneous multi-agent systems. By combining type-specific graph attention with information-theoretic message refinement, it enables diverse agents to coordinate effectively in complex environments. Future work will explore dynamic role discovery and the application of CLAR to real-world heterogeneous systems like urban traffic management.

Summary updated

BibTeX
@inproceedings{du2023heterogeneous,
  author = {Wei Du and Shifei Ding},
  title = {{Heterogeneous Multi-Agent Communication Learning via Graph Information Maximization}},
  booktitle = {SEKE},
  year = {2023},
  url = {https://openreview.net/forum?id=vAvpgwF4ri},
}
Figure