Abstract
Multi-agent reinforcement learning (MARL) is often hindered by partial observability and poor scalability in large agent populations. Existing frameworks like Value Function Decomposition (VFD) are effective for cooperative tasks but struggle in non-cooperative settings, while Mean-Field Reinforcement Learning (MFRL) simplifies interactions but often requires global information. We introduce Mean-Field Value Function Decomposition (MFVFD), which bridges these approaches by decomposing individual joint Q-functions using an attention-based mean-field approximation. MFVFD enables decentralized execution while capturing complex local interactions during centralized training. We evaluate MFVFD across matrix games, cooperative navigation, and large-scale battles with up to 800 agents. Results show that MFVFD consistently outperforms state-of-the-art baselines in performance, scalability, and convergence speed.
Introduction
The primary challenges in MARL are the exponential growth of the joint action space and the limitations of partial observability. Current solutions typically fall into two categories: VFD methods (e.g., VDN, QMIX), which assume a shared team reward and fail in competitive environments, and MFRL methods, which approximate population effects but often assume agent homogeneity. MFVFD addresses these limitations by providing a scalable architecture that handles cooperative, competitive, and mixed tasks through individual-centric decomposition and decentralized execution.
Methods
MFVFD decomposes an agent's individual joint Q-function into two components: a local value ($Q_{LOC}$) based on private observations and actions, and a mean-field value ($Q_{MF}$) representing the influence of neighboring agents. To model these interactions precisely, we employ an attention mechanism that uses a query-key system to weight the importance of specific neighbors. This allows the model to focus on the most relevant local interactions. MFVFD follows the Centralized Training and Decentralized Execution (CTDE) paradigm, ensuring practical deployment.
Experiments and Results
We evaluated MFVFD in several environments to test its versatility and scalability:
- Matrix Games: In the non-cooperative 'Hawk-Dove' game, MFVFD successfully converged to the Nash Equilibrium, whereas cooperative baselines like VDN failed to account for non-cooperative stable strategies.
- Cooperative Navigation: MFVFD demonstrated significantly faster convergence and higher mean episode rewards than QMIX, SAC, and MADDPG.
- MAgent Battle: In a large-scale scenario with 800 agents, MFVFD outperformed DQN, Actor-Critic, and existing mean-field methods (MFQ/MFAC).
- Flow Traffic Control: MFVFD achieved higher rewards in both Ring and Figure-Eight benchmarks compared to VDN and DQN.
Discussion and Conclusion
MFVFD offers a robust framework for MARL by integrating the scalability of mean-field theory with the structural advantages of value function decomposition. By focusing on individual-centric Q-functions and utilizing attention mechanisms, it effectively manages complex agent interactions across various task types. Future work will explore scaling to environments with a high diversity of agent classes and further optimizing the attention mechanism for extreme agent densities.
Summary updated