Graph Neural Network-based Multi-agent Reinforcement Learning for Resilient Distributed Coordination of Multi-Robot Systems

Anthony Goeckner, Yueyuan Sui, Nicolas Martinet, Xinliang Li, Qi Zhu

Abstract

Deploying multi-robot systems in unpredictable real-world environments requires coordination strategies that are resilient to hardware failure and communication degradation. We introduce MAGEC (Multi-Agent Graph Embedding-based Coordination), a reinforcement learning framework that leverages Graph Neural Networks (GNNs) to enable robust swarm coordination. By modeling environments as graphs and utilizing an inductive neighbor-scoring mechanism, MAGEC maintains high efficiency even under significant robot attrition and 90% communication loss.

Introduction

The deployment of multi-robot systems for tasks such as search-and-rescue or urban patrolling is often hindered by the fragility of traditional coordination algorithms. Most existing methods struggle when individual robots fail or when communication signals become intermittent. A critical gap in current Multi-Agent Reinforcement Learning (MARL) research is the effective use of GNNs for "wayfinding"—the process of deciding the next node to visit in a graph-based environment while accounting for partial observability.

Methodology: The MAGEC Framework

MAGEC combines Multi-Agent Proximal Policy Optimization (MAPPO) with a specialized GNN architecture designed for decentralized execution.

The MAGEC architecture, showing the interaction between the centralized critic and decentralized actor networks within the environment loop.

Key Technical Innovations

  • Edge-Aware Inductive GNN: Unlike standard node-centric models, MAGEC incorporates edge features such as distance and connectivity. Its inductive nature allows the policy to generalize to entirely new maps without retraining.
  • Neighbor Scoring Mechanism: The actor network employs a "neighbor-scorer" MLP that processes graph embeddings of adjacent nodes to produce utility scores. A "selector" then chooses the optimal next move based on these scores.
  • Synchronous Step-Skipping: To handle the sparse nature of decision-making during travel between nodes, the training algorithm skips redundant time steps, enabling the use of Generalized Advantage Estimation (GAE) for delayed rewards.
  • Centralized Training, Decentralized Execution (CTDE): A global "critic" assists during training, while individual "actors" make decisions based solely on local, partial views during deployment.

Experimental Evaluation

The framework was evaluated using the Patrolling Zoo environment and Grex Machina, a ROS 2-based framework for high-fidelity simulation. Agents were trained on the "Milwaukee" graph and tested on the "Cumberland" benchmark graph to evaluate zero-shot generalization.

Results and Discussion

MAGEC was benchmarked against state-of-the-art algorithms including AHPA, SEBS, and CBLS.

Resilience to Attrition and Communication Loss

A defining characteristic of MAGEC is its emergent resilience. The system was not explicitly trained for robot attrition, yet it naturally adapted to the loss of teammates during execution.

Comparison of average node idleness under robot attrition. MAGEC maintains significantly lower idleness compared to baseline algorithms as robots are removed.

Furthermore, MAGEC demonstrated remarkable stability under extreme communication constraints. Even with a communication success rate as low as 10%, the increase in node idleness was minimal compared to the sharp performance degradation seen in competing methods.

Generalization

The inductive GNN approach allowed for successful zero-shot generalization. A policy trained with four agents on one map successfully patrolled a different map with six agents, highlighting the scalability of the graph-based representation.

Conclusion

MAGEC provides a robust solution for multi-robot coordination by treating the environment as a universal graph. While highly resilient, performance does degrade when communication is completely lost (0%). Future research will focus on integrating state-prediction methods to allow agents to estimate teammate positions and node states during total communication blackouts.

Summary updated

BibTeX
@inproceedings{goeckner2024graph,
  author = {Anthony Goeckner and Yueyuan Sui and Nicolas Martinet and Xinliang Li and Qi Zhu},
  title = {{Graph Neural Network-based Multi-agent Reinforcement Learning for Resilient Distributed Coordination of Multi-Robot Systems}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802510},
}
Figure