Abstract
In Multi-Agent Reinforcement Learning (MARL), coordinating heterogeneous agents with diverse observation and action spaces remains a significant challenge. Traditional communication methods often rely on homogeneous Graph Neural Networks (GNNs) that fail to account for distinct agent roles, leading to noisy and ineffective information exchange. This paper presents CLAR (Communication Learning of multi-Agent Reinforcement learning), a framework that leverages heterogeneous graph modeling and Mutual Information (MI) maximization to facilitate effective communication among diverse agents. Experimental results on the Predator-Capture-Prey and StarCraft II Multi-Agent Challenge (SMAC) benchmarks demonstrate that CLAR significantly outperforms state-of-the-art baselines in complex, heterogeneous scenarios.
Introduction
Most existing MARL research operates under the assumption of homogeneity, where agents share identical capabilities and perspectives. However, real-world applications—ranging from heterogeneous robot swarms to complex tactical simulations—require coordination between agents with vastly different sensors and actuators. When standard GNN-based communication is applied to these systems, the indiscriminate aggregation of features often results in "communication noise," where critical role-specific information is lost.
To address this, the researchers from China University of Mining and Technology developed CLAR. The framework focuses on two primary objectives: making communication type-aware through heterogeneous graph attention and ensuring message relevance through information theory.
Methodology
The CLAR architecture is structured into three distinct layers: feature encoding, communication, and value decomposition.
1. Feature Encoding
Each agent $i$ processes its local observation $o_i$ through an MLP and a Gated Recurrent Unit (GRU) to generate a hidden state $h_i$. This state captures the temporal dependencies of the agent's experience. To support heterogeneous communication, an additional encoder transforms these hidden states into class-specific feature representations.
2. Heterogeneous Communication via HGNN
Instead of treating all neighbors equally, CLAR utilizes a Heterogeneous Graph Attention Network (HGAT). This module employs type-specific weight matrices for different "edge types" (e.g., a message from a 'scout' to an 'attacker'). This allows the network to learn unique relational dynamics between different roles, ensuring that agents can filter and prioritize information based on the sender's class.
3. Mutual Information (MI) Maximization
A core innovation of CLAR is the use of MI maximization to refine communication. To prevent the transmission of irrelevant data, the authors introduce an MI loss function. By maximizing the correlation between received messages $m_i$ and the agent's local action-value $Q_i$, the system forces the communication module to produce "expressive" messages that are directly beneficial for decision-making. This is implemented using a discriminator that distinguishes between true message-value pairs and shuffled ones.
4. Value Factorization
CLAR follows the Centralized Training, Decentralized Execution (CTDE) paradigm. It integrates a mixing network (similar to QMIX or QPLEX) that decomposes the joint team reward into individual agent values, ensuring stable training while allowing agents to act independently during execution based on their local observations and received messages.
Experimental Evaluation
The framework was evaluated on two challenging benchmarks:
- Predator-Capture-Prey (PCP): A task requiring coordination between "predators" (who observe the environment) and "captures" (who are blind but must catch the prey based on predator instructions).
- StarCraft II Multi-Agent Challenge (SMAC): Tests were conducted on heterogeneous maps such as
2c3s5z(2 Colossi, 3 Stalkers, 5 Zealots) andMMM2.
Key Results
CLAR demonstrated substantial performance gains across all tested environments. In the highly complex 2c3s5z map of SMAC, CLAR achieved a peak win rate of approximately 93%, whereas standard QMIX reached only ~78% and the communication-based TarMAC reached ~60%.
Ablation studies confirmed the necessity of both the heterogeneous modeling and the MI optimization. Removing the MI loss (CLAR-M) or the heterogeneous GNN components (CLAR-H) resulted in significant performance degradation, proving that role-aware filtering and message relevance are both critical for heterogeneous coordination.
Furthermore, visualization of the message space using t-SNE revealed that MI optimization leads to distinct clustering of agent communications based on their functional roles. This indicates that the agents successfully learned a specialized "language" tailored to their specific tasks within the team.
Conclusion
CLAR provides a robust solution for the communication challenges inherent in heterogeneous multi-agent systems. By combining type-specific graph attention with information-theoretic message refinement, it enables diverse agents to coordinate effectively in complex environments. Future work will explore dynamic role discovery and the application of CLAR to real-world heterogeneous systems like urban traffic management.
Summary updated