Promoting Coordination Through Electing First-moveAgent in Multi-Agent Reinforcement Learning

Jingqing Ruan, Linghui Meng, Bo Xu

Introduction

In traditional Multi-Agent Reinforcement Learning (MARL), agents typically make decisions simultaneously. While computationally efficient, this "simultaneous action" norm often leads to coordination failures in complex environments. For instance, in a soccer match, two players might simultaneously attempt to tackle the same opponent, leaving other areas undefended. Existing solutions often rely on implicit coordination through communication or random asynchronous execution, neither of which fully captures the strategic necessity of situational leadership.

The EFA-DWM Framework

This paper introduces EFA-DWM, a framework designed to break the simultaneous action paradigm by electing a "first-move" leader dynamically. The architecture consists of two primary components: the Electing First-move Agent (EFA) module and the Dynamically Weighted Mixing (DWM) network.

Overview of the EFA-DWM architecture, showing the interaction between the EFA module, DWM module, and the training process.

Electing First-move Agent (EFA) Module

The EFA module serves as the core of the decision-making hierarchy. It processes local observations and previous actions through a Gated Recurrent Unit (GRU) based Message Encoder. These encoded features are then passed to a Message Aggregator that utilizes a Graph Convolutional Network (GCN) with multi-head attention. This allows agents to model their spatial and functional relationships as a dynamic graph. Finally, a Weight Generator outputs a weight for each agent, and the agent with the highest weight—determined via a differentiable Gumbel-Softmax estimator—is elected as the leader.

Dynamically Weighted Mixing (DWM) Network

Once the leader is chosen and acts, the remaining agents respond. To estimate the value of these joint actions, the authors propose the DWM network. Building upon Value Decomposition Networks (VDN), DWM introduces a dynamic penalty factor ($\alpha$). This factor adjusts weights during training to prioritize states where joint action values are underestimated and penalize overestimation, leading to more stable and accurate value function learning.

Experimental Evaluation

The researchers evaluated EFA-DWM on two benchmarks: Cooperative Navigation (MPE) and Google Research Football (GRF). In MPE, the system was tested with varying numbers of agents (2, 3, and 5). In the more complex GRF environment, two scenarios were prioritized: a 3-vs-1 setup and a challenging 2-vs-6 "less-versus-more" scenario.

Key Results

EFA-DWM consistently outperformed popular baselines such as QMIX, VDN, and G2ANet across all tested environments. In Cooperative Navigation, the model achieved higher rewards with lower variance, indicating more stable learning.

Average episode rewards in Google Research Football scenarios (3-vs-1 and 2-vs-6), showing EFA-DWM outperforming standard MARL baselines.

The performance gap was most notable in the Google Football 2-vs-6 scenario. As shown in the results, EFA-DWM significantly outperformed all baselines in the later stages of training. This suggests that the hierarchical approach is particularly effective in environments where precise coordination is the only viable path to success. Ablation studies further confirmed that both the EFA election mechanism and the DWM weighted mixing are essential for these performance gains.

Conclusion

By explicitly modeling situational leadership through the EFA-DWM framework, this research demonstrates that asynchronous, hierarchical decision-making can significantly enhance coordination in MARL. While the current model focuses on a single leader, future work may explore multi-level hierarchies to scale these benefits to even larger and more heterogeneous multi-agent teams.

Summary updated

BibTeX
@inproceedings{ruan2022promoting,
  author = {Jingqing Ruan and Linghui Meng and Bo Xu},
  title = {{Promoting Coordination Through Electing First-moveAgent in Multi-Agent Reinforcement Learning}},
  booktitle = {ICAPS},
  year = {2022},
  url = {https://openreview.net/forum?id=4eXEpxyUQIH},
}
Figure