Resilient Multi-Agent Reinforcement Learning with Adversarial Value Decomposition

Thomy Phan, Lenz Belzner, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, Claudia Linnhoff-Popien

Introduction

In cooperative Multi-Agent Reinforcement Learning (MARL), agents are typically trained to achieve a shared objective. However, standard approaches often lead to "overfitting" to specific teammate behaviors. In real-world deployments—such as robotic swarms or industrial automation—hardware failures or software updates can alter an agent's performance. When these changes occur, traditional MARL systems frequently collapse because they assume teammates will behave exactly as they did during training.

Problem Statement

The core challenge is developing agents that are resilient to unexpected changes in their partners' behaviors. While some existing methods address resilience, they often focus on extreme edge cases or require precise, pre-defined failure rates during training. There is a need for a framework that provides robustness against a spectrum of teammate behaviors without intensive hyperparameter tuning.

Methodology: RADAR

The proposed framework, RADAR (Resilient Adversarial value Decomposition with Antagonist-Ratios), addresses these challenges through two primary innovations:

Randomized Adversarial Training (RAT)

Instead of training with a static team, RAT iteratively samples an "antagonist ratio" ($R_{ant}$) from a uniform distribution. This creates mixed-game scenarios where some agents act as "protagonists" (task-oriented) and others as "antagonists" (adversarial). This diversity forces protagonists to learn robust strategies that succeed even when teammates underperform or actively hinder progress.

Adversarial Value Decomposition

To integrate RAT with Centralized Training, Decentralized Execution (CTDE) frameworks, RADAR utilizes a specialized value decomposition scheme based on Value Decomposition Networks (VDN).

The RADAR framework architecture, illustrating the Randomized Adversarial Training (RAT) process and the dual-VDN value decomposition.

As shown in the architecture diagram, the system maintains separate VDN instances for protagonist and antagonist teams. Crucially, team returns are normalized by the number of active agents, allowing the network to handle variable team sizes—a significant advancement over previous MARL algorithms that required fixed input dimensions.

Experimental Evaluation

The researchers evaluated RADAR across two domains:

  1. Predator-Prey (PP): A gridworld task requiring coordination to capture moving prey, tested with 4 and 8 agents.
  2. Cyber-Physical Production System (CPPS): A complex industrial simulation where agents must navigate tasks through workstations. This environment features bottlenecks and hubs, making coordination essential to prevent deadlocks.

Results and Discussion

RADAR demonstrates superior resilience compared to state-of-the-art models like COMA, QMIX, and M3DDPG.

Performance comparison of RADAR against baselines in the CPPS environment across different agent counts and antagonist ratios.

Key findings include:

  • Worst-Case Robustness: RADAR consistently outperformed competitors in scenarios involving unknown adversaries at test time. In the 16-agent CPPS task, its performance was nearly double that of baseline methods.
  • Efficiency in Cooperation: Unlike many robust methods that suffer a "cooperation tax," RADAR remains highly competitive in purely cooperative settings.
  • Graceful Degradation: RADAR-trained agents exhibit a slower decline in performance as the number of failing teammates increases, whereas standard MARL agents often fail completely after a single teammate's behavior changes.

Conclusion

RADAR provides a "plug-and-play" solution for resilient MARL by combining randomized adversarial training with a flexible value decomposition scheme. By leveraging the linear nature of VDN for normalization, it handles variable team sizes effectively without requiring complex hyperparameter tuning. Future work may explore extending these principles to non-linear decomposition methods like QMIX or implementing adaptive sampling strategies to further optimize the training process.

Summary updated

Abstract

We focus on resilience in cooperative multi-agent systems, where agents can change their behavior due to udpates or failures of hardware and software components. Current state-of-the-art approaches to cooperative multi-agent reinforcement learning (MARL) have either focused on idealized settings without any changes or on very specialized scenarios, where the number of changing agents is fixed, e.g., in extreme cases with only one productive agent. Therefore, we propose Resilient Adversarial value Decomposition with Antagonist-Ratios (RADAR). RADAR offers a value decomposition scheme to train competing teams of varying size for improved resilience against arbitrary agent changes. We evaluate RADAR in two cooperative multi-agent domains and show that RADAR achieves better worst case performance w.r.t. arbitrary agent changes than state-of-the-art MARL.
BibTeX
@inproceedings{phan2021resilient,
  author = {Thomy Phan and Lenz Belzner and Thomas Gabor and Andreas Sedlmeier and Fabian Ritz and Claudia Linnhoff-Popien},
  title = {{Resilient Multi-Agent Reinforcement Learning with Adversarial Value Decomposition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i13.17348},
}
Figure