Dynamic Heterogeneous Multi-Agent Inverse Reinforcement Learning Based on Graph Attention Mean Field

Li Song, Irfan Ali Channa, Zeyu Wang, Guangyu SunSymmetry 2025 · DOI

TL;DR

This paper introduces GAMF-DHIRL, a novel framework for multi-agent inverse reinforcement learning (MA-IRL) that can infer reward functions in dynamic, heterogeneous environments. By combining graph attention, mean-field theory, and adversarial learning, it effectively models complex agent interactions and adapts to changing agent behaviors, outperforming existing methods.

What Problem Does This Paper Address?

Traditional multi-agent inverse reinforcement learning (MA-IRL) methods often struggle with real-world scenarios where agents are not only heterogeneous (having different types, states, and objectives) but also interact in dynamic environments with changing relationships (topologies). Most existing algorithms are designed for homogeneous agents and static interaction structures, making them ill-suited for applications like autonomous driving, drone swarms, or robot clusters. This paper aims to bridge this gap by developing an MA-IRL framework that can handle dynamic topologies and heterogeneous agent behaviors, enabling more robust and adaptable reward inference.

Methods

The core of the proposed solution is the Graph Attention Mean Field Dynamic Heterogeneous Inverse Reinforcement Learning (GAMF-DHIRL) framework. Key components include:

  • Graph Attention Mechanism: This is used to dynamically model dependencies between agents. It adaptively adjusts the influence weights of neighboring nodes through attention, allowing the model to focus on relevant interactions.
  • Mean Field Theory: To manage the complexity of multi-agent interactions, a typed mean-field approximation is employed. This approach models the collective behavior of groups of agents, reducing computational burden while still capturing essential group dynamics.
  • Adversarial Inverse Reinforcement Learning (AIRL): Building upon the AIRL paradigm, the framework uses a generator-discriminator setup. The discriminator learns to distinguish between expert demonstrations and generated trajectories, while the generator (policy) learns to produce trajectories that fool the discriminator. This adversarial process helps in recovering reward functions without the ambiguity problem common in IRL.
  • Dynamic Adjacency Matrix: To capture the time-varying nature of agent interactions, the GAMF module utilizes a dynamic adjacency matrix.
  • Online Type Inference and Clustering: The framework incorporates an online EM algorithm to dynamically identify and update agent types and their corresponding prototypes, allowing adaptation to changing agent behaviors.

Datasets

The paper uses simulated particle environments for evaluation. Specifically, it tests the algorithm in three scenarios:

  • Cooperative Navigation: Agents navigate a 2D space to cover dynamic targets while avoiding collisions.
  • Predator-Prey: A classic adversarial scenario simulating predators hunting prey.
  • Cooperative Communication: Agents collaborate, communicate, and observe to achieve tasks, with one agent acting as a "listener."

The paper does not introduce new benchmark datasets but rather uses these established simulated environments to evaluate its proposed method against existing ones.

Key Insights and Ideas

  • Bridging Heterogeneity and Dynamics: The most significant insight is the successful integration of graph attention and mean-field theory to handle both heterogeneous agents and dynamic interaction topologies, a critical challenge in current MA-IRL.
  • Adaptive Interaction Modeling: The use of graph attention allows the model to dynamically weigh the importance of different agent interactions, crucial for scenarios where relationships change over time.
  • Computational Efficiency: The typed mean-field approximation significantly reduces computational complexity, making the approach scalable to larger numbers of agents.
  • Robust Reward Recovery: The adversarial framework, combined with the dynamic modeling, leads to more accurate recovery of reward functions, even in complex and competitive environments.

Key Results

  • Reward Recovery: GAMF-DHIRL consistently recovers reward functions that are closer to the ground truth compared to baseline methods (MA-GAIRL, MA-GAIL, MA-IRL) across all tested scenarios.
  • Performance Improvement: The paper demonstrates that GAMF-DHIRL outperforms MA-GAIRL, MA-GAIL, and MA-IRL in terms of value loss, explained variance, Q values, and reported rewards. For instance, in cooperative navigation, GAMF-DHIRL achieved a reported reward of 0.894 ± 0.004, significantly higher than MA-GAIRL's 0.805 ± 0.002.
  • Efficiency: Ablation studies show that GAMF-DHIRL has a shorter running time than MA-GAIRL and MF-AIRL, indicating that the graph attention mean field approach can reduce computational complexity.
  • Sensitivity Analysis: The study found that an average neighborhood size of 8 yielded the best Q values, suggesting an optimal balance for capturing local interactions.

Limitations and Future Work

  • Scalability to Extreme Numbers: The authors acknowledge that the interference resistance and scalability of the proposed algorithm have not been verified for extremely large numbers of agents (tens of thousands).
  • Improving Interference Resistance: Future work will focus on enhancing the algorithm's ability to handle scenarios with dynamic or moving targets, improving coverage.
  • Enhancing Scalability: Further research will explore how to maintain good performance as the number of agents increases significantly.

Summary updated

Abstract

Multi-agent inverse reinforcement learning (MA-IRL) infers the underlying reward functions or objectives of multiple agents by observing their behavioral data, thereby providing insights into collaboration, competition, or mixed interaction strategies among agents, and addressing the symmetrical ambiguity problem where multiple rewards may correspond to the same strategy. However, most existing algorithms mainly focus on solving cooperative and non-cooperative tasks among homogeneous multi-agent systems, making it difficult to adapt to the dynamic topologies and heterogeneous behavioral strategies of multi-agent systems in real-world applications. This makes it difficult for the algorithm to adapt to scenarios with locally sparse interactions and dynamic heterogeneity, such as autonomous driving, drone swarms, and robot clusters. To address this problem, this study proposes a dynamic heterogeneous multi-agent inverse reinforcement learning framework (GAMF-DHIRL) based on a graph attention mean field (GAMF) to infer the potential reward functions of agents. In GAMF-DHIRL, we introduce a graph attention mean field theory based on adversarial maximum entropy inverse reinforcement learning to dynamically model dependencies between agents and adaptively adjust the influence weights of neighboring nodes through attention mechanisms. Specifically, the GAMF module uses a dynamic adjacency matrix to capture the time-varying characteristics of the interactions among agents. Meanwhile, the typed mean-field approximation reduces computational complexity. Experiments demonstrate that the proposed method can efficiently recover reward functions of heterogeneous agents in collaborative tasks and adversarial environments, and it outperforms traditional MA-IRL methods.
BibTeX
@inproceedings{song2025dynamic,
  author = {Li Song and Irfan Ali Channa and Zeyu Wang and Guangyu Sun},
  title = {{Dynamic Heterogeneous Multi-Agent Inverse Reinforcement Learning Based on Graph Attention Mean Field}},
  booktitle = {Symmetry},
  year = {2025},
  doi = {10.3390/sym17111951},
}
Figure