Abstract
Recent advancements in multi-agent reinforcement learning (MARL) have opened up vast application prospects, such as swarm control of drones, collaborative manipulation by robotic arms, and multi-target encirclement. However, potential security threats during the MARL deployment need more attention and thorough investigation. Recent research reveals that attackers can rapidly exploit the victim's vulnerabilities, generating adversarial policies that result in the failure of specific tasks. Existing studies predominantly focus on two-player competitive environments, assuming attackers possess complete global state observation. In this study, we unveil, for the first time, the capability of attackers to generate adversarial policies even when restricted to partial observations of the victims in multi-agent competitive environments. Specifically, we propose a novel black-box attack (SUB-PLAY) that incorporates the concept of constructing multiple subgames to mitigate the impact of partial observability and suggests sharing transitions among subpolicies to improve attackers' exploitative ability. Extensive evaluations demonstrate the effectiveness of SUB-PLAY under three typical partial observability limitations. Visualization results indicate that adversarial policies induce significantly different activations of the victims' policy networks. Furthermore, we evaluate three potential defenses aimed at exploring ways to mitigate security threats posed by adversarial policies, providing constructive recommendations for deploying MARL in competitive environments.
Introduction
While Multi-Agent Reinforcement Learning (MARL) has shown incredible promise in fields like drone swarm control and robotic collaboration, its security remains a significant concern. Previous research into adversarial policies—where an agent learns a strategy specifically designed to trigger vulnerabilities in a victim's policy—often assumed the attacker possessed global observability. However, in real-world deployments, sensors have inherent limits: LiDAR has a maximum range, obstacles cause occlusions, and measurements are frequently noisy. This study addresses whether an attacker can still successfully exploit a multi-agent system when restricted to a partial, local view of the victims. We find that standard attack methods often fail under partial observability, leading to the development of SUB-PLAY to overcome these visibility gaps.
The SUB-PLAY Framework
SUB-PLAY is a "divide-and-conquer" black-box attack designed to exploit MARL systems under realistic partial observability constraints. The architecture is built upon four main pillars:
- Subgame Construction: The attacker identifies different "visibility states" (e.g., seeing 0, 1, or $N$ victim agents) and models each as a separate Partially Observable Stochastic Game (POSG).
- Transition Dissemination: To address data imbalance where some visibility states occur more frequently than others, SUB-PLAY shares experience data (transitions) between subgame replay buffers based on their proximity.
- Subpolicy Training: Each subgame trains its own specific subpolicy using Policy Meritocracy, retaining only the best-performing versions to mitigate non-stationarity.
- Policy Combination: During deployment, the attacker uses a hard-coded switch to instantly swap to the corresponding subpolicy based on the number of currently visible victim agents.
Experimental Evaluation
The effectiveness of SUB-PLAY was evaluated using the OpenAI Multi-Agent Particle Environments (MPE), focusing on Predator-Prey and World Communication tasks. These tasks were tested across various agent configurations and three specific observability limitations: Uncertainty (random noise), Distance (limited sensor range), and Region (geographic occlusions).
Results and Insights
Extensive evaluations demonstrate that SUB-PLAY significantly degrades victim performance, reducing it to approximately 52-60% of the baseline across all tested limitations. Compared to the previous state-of-the-art (Victim-play), SUB-PLAY showed an average improvement in attack effectiveness of 27% to 50%.
Key insights include:
- Heterogeneity and Data Sharing: The "Occupancy Rate" of visibility states varies wildly. The transition dissemination mechanism is crucial for training subpolicies for rare states.
- Curriculum Learning via Masking: In complex scenarios, partial observability can act as a natural curriculum, simplifying the initial learning task.
- Neural Anomalies: t-SNE visualizations revealed that adversarial policies force the victim's policy network into activation states never encountered during normal training, effectively "paralyzing" the victim's decision-making.
Discussion and Future Work
While effective, SUB-PLAY faces challenges such as the computational overhead of training multiple subpolicies and the requirement for extensive interaction with the victim during training. Regarding defense, standard adversarial retraining appears insufficient; instead, Policy Ensembles are recommended to prevent attackers from exploiting single vulnerabilities. Future work may explore Offline RL to mitigate training requirements.
Summary updated