SUB-PLAY: Adversarial Policies against Partially Observed Multi-Agent Reinforcement Learning Systems

Oubo Ma, Yuwen Pu, Linkang Du, Yang Dai, Ruo Wang, Xiaolei Liu, Yingcai Wu, Shouling Ji

TL;DR

Adversarial policies can severely disrupt multi-agent reinforcement learning (MARL) systems even when the attacker's vision is heavily restricted. The authors introduce SUB-PLAY, a black-box attack framework that uses a divide-and-conquer strategy to break down complex, partially observable environments into manageable subgames, exposing critical security vulnerabilities in deployments like drone swarms or robotic teams.

What Problem Does This Paper Address?

While Multi-Agent Reinforcement Learning (MARL) has driven breakthroughs in autonomous driving, drone swarms, and robotics, its security vulnerabilities remain dangerously understudied. Existing research shows that attackers can train "adversarial policies"—deploying agents that behave bizarrely to exploit a victim's blind spots and force them into catastrophic failures.

However, prior attack frameworks assumed a major luxury: that the attacker has perfect, global knowledge of the environment. In real-world deployments, physical constraints create partial observability. Attackers usually suffer from sensor noise, limited radar range, or physical obstacles.

Figure 1: Three partially observable limitations in multi- agent environments.

This paper asks a critical question: Can an attacker still successfully exploit a multi-agent system if they can only see a fraction of what is happening?

Methods

The authors propose SUB-PLAY, a novel black-box attack framework designed to overcome the challenges of partial observability through a divide-and-conquer approach. It operates via four main steps:

  1. Subgame Construction: Instead of tackling one massive, partially observable game, SUB-PLAY decomposes the environment into multiple simpler subgames based on the number of victim agents currently observed.
  2. Transition Dissemination: A major challenge in training subpolicies is data scarcity; some subgames occur much less frequently than others. To prevent undertraining, SUB-PLAY introduces a mechanism that shares experience transitions between geometrically close subgames.
  3. Policy Meritocracy: To handle the non-stationary nature of multi-agent environments (where policies constantly change), the framework uses a selection pool based on the harmonic mean of multiple test metrics, ensuring only the most robust subpolicies are retained.
  4. Policy Combination: During deployment, the attacker uses a hard-coded switching mechanism. Depending on how many victim agents are visible at any given time step, the adversary instantly swaps to the corresponding optimized subpolicy.

Crucially, SUB-PLAY is algorithm-agnostic, meaning it works effectively against victims trained via both distributed and centralized MARL paradigms.

Figure 2: MARL training paradigms.

Datasets

The framework was evaluated using OpenAI's Multi-Agent Particle Environments (MPE), focusing on two competitive multi-agent tasks:

  • Predator-prey: A benchmark where slower predators (the victims) must cooperate to catch faster prey (the adversaries).
  • World Communication: A more complex environment featuring food elements, a forest that completely obstructs visibility, and a specialized "leader" predator with communication capabilities.

The attacks were systematically tested across various agent configurations, including 1v3, 2v3, 3v3, 2v2, and 4v2 scenarios.

Key Insights and Ideas

  • Partial Observability is No Shield: MARL defenders cannot rely on an attacker's limited vision for security. By focusing purely on maximizing their own rewards within local subgames, attackers can still reliably trigger global failures in the victim's policy.
  • The Power of Proactive Masking: Counterintuitively, intentionally hiding parts of the environment's observations from the attacker during early training can actually improve attack performance. This "proactive masking" simplifies the initial input space, accelerating early policy convergence.
  • Defenses are Computationally Intractable: Standard fixes like adversarial retraining fail to neutralize the threat. Because finding a true Nash Equilibrium in complex multi-agent setups is a known PPAD-complete problem, exploitable policy gaps will almost always remain.

Key Results

  • Substantial Performance Degradation: Across extensive testing, SUB-PLAY consistently outperformed standard attack baselines. Under uncertainty limitations, it reduced the victim's performance to 51.98% of its normal baseline. Under distance-based sight limitations, performance dropped to 55.71%.
  • Network Activation Anomalies: Using t-SNE visualizations, the authors demonstrated that SUB-PLAY forces completely different activation patterns within the victim's neural networks compared to normal opponents, proving that the victim is successfully manipulated into uncharacteristic, suboptimal decision-making states.

Limitations and Future Work

While highly effective, SUB-PLAY has a few limitations. First, due to the scarcity of standardized, partially observable multi-agent competitive benchmarks, it was only validated within the MPE framework. Second, training separate subpolicies increases computational costs linearly with the number of subgames constructed.

Finally, the current framework assumes the attacker can interact with the victim dynamically during the training phase. The authors note that exploring offline RL techniques—allowing attackers to generate adversarial policies purely from pre-recorded datasets—is a promising direction for future work.

Summary updated

Abstract

Recent advancements in multi-agent reinforcement learning (MARL) have opened up vast application prospects, such as swarm control of drones, collaborative manipulation by robotic arms, and multi-target encirclement. However, potential security threats during the MARL deployment need more attention and thorough investigation. Recent research reveals that attackers can rapidly exploit the victim's vulnerabilities, generating adversarial policies that result in the failure of specific tasks. For instance, reducing the winning rate of a superhuman-level Go AI to around 20%. Existing studies predominantly focus on two-player competitive environments, assuming attackers possess complete global state observation. In this study, we unveil, for the first time, the capability of attackers to generate adversarial policies even when restricted to partial observations of the victims in multi-agent competitive environments. Specifically, we propose a novel black-box attack (SUB-PLAY) that incorporates the concept of constructing multiple subgames to mitigate the impact of partial observability and suggests sharing transitions among subpolicies to improve attackers' exploitative ability. Extensive evaluations demonstrate the effectiveness of SUB-PLAY under three typical partial observability limitations. Visualization results indicate that adversarial policies induce significantly different activations of the victims' policy networks. Furthermore, we evaluate three potential defenses aimed at exploring ways to mitigate security threats posed by adversarial policies, providing constructive recommendations for deploying MARL in competitive environments.
BibTeX
@inproceedings{ma2024sub,
  author = {Oubo Ma and Yuwen Pu and Linkang Du and Yang Dai and Ruo Wang and Xiaolei Liu and Yingcai Wu and Shouling Ji},
  title = {{SUB-PLAY: Adversarial Policies against Partially Observed Multi-Agent Reinforcement Learning Systems}},
  booktitle = {CCS},
  year = {2024},
  doi = {10.1145/3658644.3670293},
}
Figure