Abstract
In cooperative multi-agent tasks, a team of agents jointly interact with an environment by taking actions, receiving a team reward and observing the next state. During the interactions, the uncertainty of environment and reward will inevitably induce stochasticity in the long-term returns and the randomness can be exacerbated with the increasing number of agents. However, such randomness is ignored by most of the existing value-based multi-agent reinforcement learning (MARL) methods, which only model the expectation of Q-value for both individual agents and the team. Compared to using the expectations of the long-term returns, it is preferable to directly model the stochasticity by estimating the returns through distributions. With this motivation, this work proposes a novel value-based MARL framework from a distributional perspective, i.e., parameterizing value function via Mixture of Categorical distributions for MARL. Specifically, we model both individual Q-values and global Q-value with categorical distribution. To integrate categorical distributions, we define five basic operations on the distribution, which allow the generalization of expected value function factorization methods (e.g., VDN and QMIX) to their MCMARL variants. We further prove that our MCMARL framework satisfies Distributional-Individual-Global-Max (DIGM) principle with respect to the expectation of distribution, which guarantees the consistency between joint and individual greedy action selections in the global Q-value and individual Q-values. Empirically, we evaluate MCMARL on both a stochastic matrix game and a challenging set of StarCraft II micromanagement tasks, showing the efficacy of our framework.
Introduction
In cooperative Multi-Agent Reinforcement Learning (MARL), a team of agents must coordinate to maximize a shared reward. Traditional state-of-the-art methods like QMIX and VDN rely on value function factorization, decomposing the team's global Q-value into individual agent Q-values. However, these methods typically only model the expectation (the mean) of the return.
Real-world environments are inherently stochastic due to partial observability, noisy rewards, and unpredictable agent interactions. Modeling only the mean ignores the variance and the complex "shape" of potential outcomes. This work introduces MCMARL (Mixture of Categorical distributions for MARL), a framework that models the full distribution of returns. By doing so, it enables agents to account for environmental uncertainty more effectively while maintaining the Distributional-Individual-Global-Max (DIGM) principle, ensuring that team-level optimal actions remain consistent with individual-level decisions.
The MCMARL Framework
Instead of a single scalar value, each agent in the MCMARL framework outputs a categorical distribution—a set of discrete "atoms" representing potential return values and their associated probabilities. To integrate these individual distributions into a coherent global team distribution, the authors define five core operations:
- Weighting: Scaling a distribution by a non-negative weight.
- Bias: Shifting a distribution by a constant.
- Convolution: Combining distributions to model the sum of independent random variables.
- Projection: Mapping distributions back onto a fixed set of atoms to maintain computational efficiency.
- Function: Applying non-linear, monotonic functions to the distribution.
By combining these operations, the authors develop DQMIX, a distributional variant of QMIX. DQMIX uses a mixing network parameterized by a hypernetwork to integrate individual distributions into a global one, mathematically preserving the DIGM principle.
Experimental Evaluation
The researchers evaluated MCMARL on two primary benchmarks: a Stochastic Matrix Game designed to test distribution recovery, and the StarCraft Multi-Agent Challenge (SMAC), a high-complexity environment for MARL.
Key Results
- Distributional Accuracy: In the matrix game, DQMIX successfully recovered complex return distributions. While traditional methods only capture the mean, DQMIX accurately modeled bimodal distributions, proving its ability to handle high-variance scenarios.
- Superior Performance on SMAC: DQMIX consistently outperformed standard QMIX and other distributional baselines. On the challenging
corridormap, DQMIX achieved a win rate of approximately 65%, whereas standard QMIX failed to achieve any wins. - Robustness and Convergence: Across various maps (e.g.,
10m_vs_11m,2s3z), MCMARL variants showed faster convergence and higher final win rates than their non-distributional counterparts.
Discussion and Future Work
The success of DQMIX highlights the "expectation trap": modeling only the mean is fundamentally limited in stochastic settings where different individual distributions can lead to the same team expectation but vastly different risk profiles. MCMARL provides a principled way to model the "sum of randomness" through convolution and projection.
A current limitation is the lack of a mathematically sound multiplication operator for categorical distributions in this context. This prevents the framework's application to attention-based models like QPLEX. Future research will focus on defining such an operator to extend distributional MARL to even more sophisticated neural architectures.
Summary updated