MFVFD: A Multi-Agent Q-Learning Approach to Cooperative and Non-Cooperative Tasks

Tianhao Zhang, Qiwei Ye, Jiang Bian, Guangming Xie, Tie-Yan Liu

Abstract

Multi-agent reinforcement learning (MARL) is often hindered by partial observability and poor scalability in large agent populations. Existing frameworks like Value Function Decomposition (VFD) are effective for cooperative tasks but struggle in non-cooperative settings, while Mean-Field Reinforcement Learning (MFRL) simplifies interactions but often requires global information. We introduce Mean-Field Value Function Decomposition (MFVFD), which bridges these approaches by decomposing individual joint Q-functions using an attention-based mean-field approximation. MFVFD enables decentralized execution while capturing complex local interactions during centralized training. We evaluate MFVFD across matrix games, cooperative navigation, and large-scale battles with up to 800 agents. Results show that MFVFD consistently outperforms state-of-the-art baselines in performance, scalability, and convergence speed.

Introduction

The primary challenges in MARL are the exponential growth of the joint action space and the limitations of partial observability. Current solutions typically fall into two categories: VFD methods (e.g., VDN, QMIX), which assume a shared team reward and fail in competitive environments, and MFRL methods, which approximate population effects but often assume agent homogeneity. MFVFD addresses these limitations by providing a scalable architecture that handles cooperative, competitive, and mixed tasks through individual-centric decomposition and decentralized execution.

Methods

MFVFD decomposes an agent's individual joint Q-function into two components: a local value ($Q_{LOC}$) based on private observations and actions, and a mean-field value ($Q_{MF}$) representing the influence of neighboring agents. To model these interactions precisely, we employ an attention mechanism that uses a query-key system to weight the importance of specific neighbors. This allows the model to focus on the most relevant local interactions. MFVFD follows the Centralized Training and Decentralized Execution (CTDE) paradigm, ensuring practical deployment.

The MFVFD Architecture: (a) Decentralized execution using local observations; (b) Attention-based mean-field calculation used during centralized training.

Experiments and Results

We evaluated MFVFD in several environments to test its versatility and scalability:

  • Matrix Games: In the non-cooperative 'Hawk-Dove' game, MFVFD successfully converged to the Nash Equilibrium, whereas cooperative baselines like VDN failed to account for non-cooperative stable strategies.
  • Cooperative Navigation: MFVFD demonstrated significantly faster convergence and higher mean episode rewards than QMIX, SAC, and MADDPG.
  • MAgent Battle: In a large-scale scenario with 800 agents, MFVFD outperformed DQN, Actor-Critic, and existing mean-field methods (MFQ/MFAC).
  • Flow Traffic Control: MFVFD achieved higher rewards in both Ring and Figure-Eight benchmarks compared to VDN and DQN.
Learning curves in the Cooperative Navigation task. MFVFD (red) shows faster convergence and higher final mean rewards compared to baselines like VDN, SAC, and QMIX.

Discussion and Conclusion

MFVFD offers a robust framework for MARL by integrating the scalability of mean-field theory with the structural advantages of value function decomposition. By focusing on individual-centric Q-functions and utilizing attention mechanisms, it effectively manages complex agent interactions across various task types. Future work will explore scaling to environments with a high diversity of agent classes and further optimizing the attention mechanism for extreme agent densities.

Summary updated

Abstract

Value function decomposition (VFD) methods under the popular paradigm of centralized training and decentralized execution (CTDE) have promoted multi-agent reinforcement learning progress. However, existing VFD methods proceed from a group's value function decomposition to only solve cooperative tasks. With the individual value function decomposition, we propose MFVFD, a novel multi-agent Q-learning approach for solving cooperative and non-cooperative tasks based on mean-field theory. Our analysis on the Hawk-Dove and Nonmonotonic Cooperation matrix games evaluate MFVFD's convergent solution. Empirical studies on the challenging mixed cooperative-competitive tasks where hundreds of agents coexist demonstrate that MFVFD significantly outperforms existing baselines.
BibTeX
@inproceedings{zhang2021mfvfd,
  author = {Tianhao Zhang and Qiwei Ye and Jiang Bian and Guangming Xie and Tie-Yan Liu},
  title = {{MFVFD: A Multi-Agent Q-Learning Approach to Cooperative and Non-Cooperative Tasks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/70},
}
Figure