Hierarchical Mean-Field Deep Reinforcement Learning for Large-Scale Multiagent Systems

Chao YuAAAI 2023 · DOI

Introduction

Multi-Agent Reinforcement Learning (MARL) faces a significant "curse of dimensionality." As the number of agents increases, the joint state-action space grows exponentially, rendering traditional coordination methods like MADDPG or VDN computationally intractable. Mean-Field Reinforcement Learning (MFRL) emerged as a potential solution by modeling interactions between an agent and the "average" effect of its neighbors. However, standard MFRL suffers from three critical limitations: the loss of individual agent importance, a gap between local and global coordination, and unrealistic execution constraints requiring knowledge of all neighbors' actions.

This paper introduces Hierarchical Mean-Field (HMF) reinforcement learning to address these challenges. HMF organizes agents into groups and employs a two-tier coordination structure—intra-group and inter-group—to enable nuanced interactions and superior scalability in systems with thousands of agents.

The HMF Framework

The core of the HMF approach is a two-level hierarchy designed to bridge the gap between individual actions and population-level objectives.

The HMF architecture featuring a two-level hierarchy with intra-group attention and inter-group supervision.

Two-Level Hierarchy

  1. Bottom Level (Intra-group): Agents are clustered into groups. Rather than simple averaging, HMF utilizes an attention mechanism to weight the influence of different neighbors. This allows the model to focus on the most relevant agents (e.g., a nearby predator) rather than treating all neighbors as equal.
  2. Top Level (Inter-group): "Virtual supervisors" represent each group. These supervisors interact to ensure that group-level behaviors are coordinated toward a global objective.

Value Function Factorization and Consistency

Each agent’s Q-value is factorized into a local component (based on individual observations) and a group-interaction component. To handle the mobility of agents in large-scale systems, the authors introduce Information and Policy Consistency (IPC). This dynamic grouping mechanism uses K-Means clustering combined with a KL-divergence penalty, ensuring that as agents transition between groups, their policies remain consistent with the new group's "mean policy."

Experimental Evaluation

The researchers evaluated HMF across four distinct environments to test coordination, scalability, and social dynamics.

Experimental environments: (a) Spread, (b) Pursuit Evasion, (c) Battle, and (d) Bar Game.
  1. Multi-Agent Particle Environments (MPE): Spread (6 agents) and Pursuit Evasion (20 agents) were used to test small-scale coordination.
  2. MAgent Battle: A large-scale environment involving 200 agents fighting 100 superior enemies, used to test scalability up to 1,400 agents.
  3. Bar Game: A classic social dilemma involving 100 agents. Agents must decide whether to attend a bar based on previous attendance, testing mixed cooperative-competitive dynamics where the optimal strategy requires careful population-level balancing.

Key Results and Insights

The HMF framework demonstrated significant improvements over baselines such as MFQ and MFVFD:

  • Superior Performance in Large-Scale Combat: In the MAgent Battle task, HMF and its IPC variant achieved higher average rewards and more stable convergence than standard Mean-Field methods.
  • Solving Social Dilemmas: While most baseline methods failed to find an optimal strategy in the Bar Game, HMF achieved near-optimal performance (0.6 reward), demonstrating its ability to handle complex population-level dynamics.
  • Massive Scalability: HMF maintained its performance advantage as the population scaled from 200 to 1,400 agents. Standard methods often degrade or become computationally intractable at these scales.
  • Execution Efficiency: By allowing agents to take actions based on local observations while benefiting from hierarchical learning, HMF is more practical for real-world deployment where global communication is often limited.

Conclusion and Future Work

HMF provides a robust framework for coordinating massive multi-agent systems by replacing simple averaging with hierarchical attention and dynamic grouping. While the Mean-Field approach remains an approximation—potentially losing information in highly heterogeneous systems—the introduction of Taylor expansion regularization helps mitigate these errors. Future work aims to apply HMF to real-world "city-level" challenges, such as autonomous fleet management and large-scale traffic control.

Summary updated

BibTeX
@inproceedings{yu2023hierarchical,
  author = {Chao Yu},
  title = {{Hierarchical Mean-Field Deep Reinforcement Learning for Large-Scale Multiagent Systems}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i10.26387},
}
Figure