Introduction
In cooperative Multi-Agent Reinforcement Learning (MARL), agents are typically trained to achieve a shared objective. However, standard approaches often lead to "overfitting" to specific teammate behaviors. In real-world deployments—such as robotic swarms or industrial automation—hardware failures or software updates can alter an agent's performance. When these changes occur, traditional MARL systems frequently collapse because they assume teammates will behave exactly as they did during training.
Problem Statement
The core challenge is developing agents that are resilient to unexpected changes in their partners' behaviors. While some existing methods address resilience, they often focus on extreme edge cases or require precise, pre-defined failure rates during training. There is a need for a framework that provides robustness against a spectrum of teammate behaviors without intensive hyperparameter tuning.
Methodology: RADAR
The proposed framework, RADAR (Resilient Adversarial value Decomposition with Antagonist-Ratios), addresses these challenges through two primary innovations:
Randomized Adversarial Training (RAT)
Instead of training with a static team, RAT iteratively samples an "antagonist ratio" ($R_{ant}$) from a uniform distribution. This creates mixed-game scenarios where some agents act as "protagonists" (task-oriented) and others as "antagonists" (adversarial). This diversity forces protagonists to learn robust strategies that succeed even when teammates underperform or actively hinder progress.
Adversarial Value Decomposition
To integrate RAT with Centralized Training, Decentralized Execution (CTDE) frameworks, RADAR utilizes a specialized value decomposition scheme based on Value Decomposition Networks (VDN).
As shown in the architecture diagram, the system maintains separate VDN instances for protagonist and antagonist teams. Crucially, team returns are normalized by the number of active agents, allowing the network to handle variable team sizes—a significant advancement over previous MARL algorithms that required fixed input dimensions.
Experimental Evaluation
The researchers evaluated RADAR across two domains:
- Predator-Prey (PP): A gridworld task requiring coordination to capture moving prey, tested with 4 and 8 agents.
- Cyber-Physical Production System (CPPS): A complex industrial simulation where agents must navigate tasks through workstations. This environment features bottlenecks and hubs, making coordination essential to prevent deadlocks.
Results and Discussion
RADAR demonstrates superior resilience compared to state-of-the-art models like COMA, QMIX, and M3DDPG.
Key findings include:
- Worst-Case Robustness: RADAR consistently outperformed competitors in scenarios involving unknown adversaries at test time. In the 16-agent CPPS task, its performance was nearly double that of baseline methods.
- Efficiency in Cooperation: Unlike many robust methods that suffer a "cooperation tax," RADAR remains highly competitive in purely cooperative settings.
- Graceful Degradation: RADAR-trained agents exhibit a slower decline in performance as the number of failing teammates increases, whereas standard MARL agents often fail completely after a single teammate's behavior changes.
Conclusion
RADAR provides a "plug-and-play" solution for resilient MARL by combining randomized adversarial training with a flexible value decomposition scheme. By leveraging the linear nature of VDN for normalization, it handles variable team sizes effectively without requiring complex hyperparameter tuning. Future work may explore extending these principles to non-linear decomposition methods like QMIX or implementing adaptive sampling strategies to further optimize the training process.
Summary updated