Abstract
FM3Q is a novel reinforcement learning framework designed to solve two-team zero-sum Markov games (2t0sMGs) by factorizing complex joint minimax Q-functions into manageable individual agent contributions. By extending the Individual-Global-Max (IGM) principle to competitive settings via the Individual-Global-MiniMax (IGMM) principle, FM3Q enables decentralized execution while maintaining strategic stability. Experimental results across various benchmarks demonstrate that FM3Q significantly outperforms existing population-based methods in both convergence speed and peak performance.
Introduction
Competitive multi-agent scenarios, such as team sports or tactical combat, are often modeled as two-team zero-sum Markov games. Solving these games requires navigating a delicate balance between intra-team cooperation and inter-team competition. Traditional approaches typically struggle with either the "combinatorial explosion" of joint action spaces when treating a team as a single super-agent or the failure of credit assignment when agents learn independently. FM3Q addresses these challenges by providing a computationally efficient framework for value decomposition in adversarial environments.
Methodology
The foundational concept of FM3Q is the Individual-Global-MiniMax (IGMM) principle. This principle ensures that the optimal joint minimax action for a team is consistent with the individual greedy actions of its members.
The FM3Q Architecture
FM3Q implements the IGMM principle through a specialized neural architecture:
- Factorized Q-Learning: The global team Q-value is decomposed into individual Q-values for each agent.
- Monotonic Mixing with Negation: A neural mixing network combines these individual values. A key innovation is the negation module applied to the opponent team's contributions, enforcing the zero-sum property where one team's gain is the other's loss.
- Centralized Training, Decentralized Execution (CTDE): While training utilizes global state information, agents execute policies independently based on local observations.
Experimental Setup
The efficacy of FM3Q was evaluated on three distinct multi-agent benchmarks:
- Wimblepong 2v2: A cooperative-competitive paddle control game.
- MPE 3v3: A physics-based environment focused on area control.
- RoboMaster 2v2: A high-fidelity robot combat simulation requiring complex coordination of movement and resources.
Results and Discussion
FM3Q demonstrated superior performance compared to established baselines such as Self-Play (SP), PSRO, and NXDO.
Learning Efficiency and Performance
As shown in the learning curves below, FM3Q exhibits significantly faster convergence. In the Wimblepong environment, FM3Q reached a 50% win rate in less than half the episodes required by competing algorithms. In the complex RoboMaster environment, FM3Q achieved a peak winning rate of approximately 0.7, substantially higher than the 0.55 achieved by the next best method, NXDO.
Strategic Stability
Beyond raw performance, FM3Q showed greater strategic stability. By utilizing a large replay buffer and avoiding the periodic data discarding common in population-based methods, FM3Q maintains a "memory" of past interactions. This prevents agents from forgetting how to counter older strategies, leading to a consistent downward trend in exploitability (measured via NashConv).
Conclusion
FM3Q successfully adapts value decomposition techniques for competitive multi-agent games. By enforcing monotonicity and zero-sum dynamics through its mixing architecture, it provides an efficient and scalable solution for 2t0sMGs. Future research will explore the incorporation of stochastic policies to reach true Nash Equilibria in games requiring randomized strategies and investigate data filtering techniques to optimize memory usage.
Summary updated