Abstract
Multi-agent reinforcement learning (MARL) methods have achieved state-of-the-art results on a range of multi-agent tasks. Yet, MARL algorithms typically require significantly more environment interactions than their single-agent counterparts to converge, a problem exacerbated by the difficulty in exploring over a large joint action space and the high variance intrinsic to MARL environments. To tackle these issues, we propose a novel algorithm that combines a decomposed centralized critic with decentralized ensemble learning, incorporating several key contributions. The main component in our scheme is a selective exploration method that leverages ensemble kurtosis. We extend the global decomposed critic with a diversity-regularized ensemble of individual critics and utilize its excess kurtosis to guide exploration toward high-uncertainty states and actions. To improve sample efficiency, we train the centralized critic with a novel truncated variation of the TD($\lambda$) algorithm, enabling efficient off-policy learning with reduced variance. On the actor side, our suggested algorithm adapts the mixed samples approach to MARL, mixing on-policy and off-policy loss functions for training the actors. This approach balances between stability and efficiency and outperforms purely off-policy learning. The evaluation shows our method outperforms state-of-the-art baselines on standard MARL benchmarks, including a variety of SMAC II maps.
Introduction
Multi-agent reinforcement learning (MARL) is notoriously data-hungry. As the number of agents increases, the joint action space expands exponentially, making random exploration highly inefficient. Furthermore, the non-stationarity of multi-agent environments introduces high variance, where a single agent's learning progress can be obscured by the changing behaviors of its teammates. Current value decomposition methods struggle to determine when to explore and which data samples are reliable for updates. This paper introduces SEMUS (Sample-Efficient Multi-Agent Actor-Critic with Uncertainty Quantification and Selective Exploration), an architecture designed to address these challenges through targeted exploration and uncertainty-aware learning.
Methodology
The SEMUS architecture is built upon four primary pillars designed to maximize sample efficiency:
- Kurtosis-Driven Selective Exploration: Unlike traditional methods that use variance to measure uncertainty, SEMUS utilizes excess kurtosis of a critic ensemble. Kurtosis is more sensitive to outliers in the ensemble's value predictions. A positive excess kurtosis signal indicates specific outlier disagreement, triggering exploration only when agents are genuinely uncertain about a state-action pair.
- Uncertainty-Aware TD($\lambda$): To mitigate the impact of noisy data, SEMUS introduces a truncated TD($\lambda$) variant. It down-weights training samples where ensemble uncertainty is high, ensuring that the centralized critic is updated using the most reliable transitions.
- Diversity-Regularized Ensemble: To prevent ensemble collapse (where all critics converge to the same weights), the model employs Bhattacharyya distance regularization. This maintains diversity among ensemble members, ensuring the uncertainty signal remains robust.
- Mixed Actor Training: The algorithm balances the stability of on-policy learning with the efficiency of off-policy data by mixing their respective loss functions during actor training.
Experimental Results
SEMUS was evaluated against state-of-the-art baselines including DOP, PAC, and QMIX across several challenging benchmarks, most notably the StarCraft Multi-Agent Challenge (SMAC).
Key Findings
- Benchmark Performance: On the "Super-Hard" MMM2 map, SEMUS achieved a win rate of approximately 81%, significantly outperforming the previous state-of-the-art, DOP, which reached only ~25%.
- Adaptation Speed: In dynamic environments where task parameters change mid-training, SEMUS demonstrated up to a 40x improvement in adaptation speed compared to standard methods.
- Robustness to Noise: In safety-critical tasks like Multi-Agent Car Following (MACF), the uncertainty-weighting mechanism allowed agents to avoid high-risk, high-uncertainty actions, leading to fewer collisions and speed violations.
Conclusion
SEMUS demonstrates that selective exploration based on higher-order statistical moments like kurtosis can significantly improve MARL sample efficiency. By focusing exploration on high-uncertainty regions and weighting training data by reliability, SEMUS achieves superior performance on complex coordination tasks. Future work will look into reducing the computational overhead of maintaining ensembles and automating hyperparameter tuning for broader applicability.
Summary updated