Adversarial Attack on Black-Box Multi-Agent by Adaptive Perturbation

Jianming Chen, Yawen Wang, Junjie Wang, Xiaofei Xie, Yuanzhe Hu, Qing Wang, Fanjiang Xu

Introduction

As multi-agent reinforcement learning (MARL) transitions from controlled research environments to real-world applications such as autonomous driving and industrial robotics, security has become a paramount concern. Most existing adversarial attack research operates under "white-box" assumptions, where the attacker has access to the model's internal weights, or assumes direct control over an agent's actions.

In practice, attackers typically face a black-box scenario, limited to observing agent inputs (observations) and outputs (actions). Furthermore, traditional attacks often target all agents or a fixed subset, making them either inefficient or easily detectable. The challenge is to develop an attack that is both effective—maximally reducing system rewards—and stealthy, utilizing minimal perturbations on a dynamically selected set of agents.

The AdapAM Framework

The authors introduce AdapAM, a framework designed to overcome black-box limitations through two primary modules:

  1. Adaptive Selection Policy: Instead of a blanket attack, AdapAM employs a hierarchical Reinforcement Learning (RL) policy based on Soft Actor-Critic (SAC). At each time step, it evaluates the environment state to determine the most "critical" victim agent and the specific "malicious action" that would inflict maximum damage.
  2. Proxy-based Perturbation: To generate effective perturbations without internal model access, the attacker trains "proxy agents" using Multi-Agent Generative Adversarial Imitation Learning (MAGAIL). These proxies mimic the target system's behavior. Once trained, a white-box C&W (Carlini & Wagner) attack is applied to the proxy to generate perturbations that successfully transfer to the black-box victim.
Overview of the AdapAM framework, illustrating the interaction between the environment, the adaptive selection policy, and the proxy-based perturbation generator.

Experimental Setup

AdapAM was evaluated across eight environments from three prominent MARL benchmarks:

  • StarCraft Multi-Agent Challenge (SMAC): Combat scenarios including 1c3s5z, 8m, and bane_vs_bane.
  • Google Research Football (GRF): Cooperative tasks like counterattack and 3_vs_1.
  • Multi-Agent Particle Environments (MPE): Navigation and communication tasks such as spread and reference.

These benchmarks provide a diverse range of agent counts and task complexities to test the framework's generalizability.

Key Insights

  • The "Critical Agent" Hypothesis: The study confirms that agent importance varies dynamically. By switching targets adaptively, AdapAM achieves superior damage with a lower perturbation budget.
  • Imitation as a Bridge: Utilizing GAIL to create functional duplicates allows attackers to convert black-box constraints into white-box opportunities through observation.
  • Stealth via Precision: AdapAM achieves stealthiness by perturbing only one agent at a time with minimal $L_\infty$ norm changes, making it difficult for standard anomaly detection to identify.

Key Results

AdapAM consistently outperformed state-of-the-art baselines like MASafe and AMCA across all tested environments. In several SMAC and Google Football scenarios, it reduced the victim's win rate from 100% to 0%.

Reward decrease comparison across various MARL benchmarks (SMAC, GRF, MPE) at different perturbation rates. AdapAM (red) consistently shows the highest impact.

The framework also demonstrated significant robustness against "Robust" MAS trained with adversarial defenses. Furthermore, in tests against anomaly detection algorithms, AdapAM's perturbations yielded the lowest F1 scores for detection, confirming its high stealthiness.

Limitations and Future Work

Despite its effectiveness, AdapAM requires substantial observation data to train proxy agents via imitation learning. Additionally, scaling the "Adaptive Selection Policy" for massive-scale swarms remains a challenge. Future research will focus on leveraging these "red teaming" insights to develop more sophisticated, adaptive defense mechanisms capable of mitigating such surgical adversarial strikes.

Summary updated

Abstract

Evaluating security and reliability for multi-agent systems (MAS) is urgent as they become increasingly prevalent in various applications. As an evaluation technique, existing adversarial attack frameworks face certain limitations, e.g., impracticality due to the requirement of white-box information or high control authority, and a lack of stealthiness or effectiveness as they often target all agents or specific fixed agents. To address these issues, we propose AdapAM, a novel framework for adversarial attacks on black-box MAS. AdapAM incorporates two key components: (1) Adaptive Selection Policy simultaneously selects the victim and determines the anticipated malicious action (the action would lead to the worst impact on MAS), balancing effectiveness and stealthiness. (2) Proxy-based Perturbation to Induce Malicious Action utilizes generative adversarial imitation learning to approximate the target MAS, allowing AdapAM to generate perturbed observations using white-box information and thus induce victims to execute malicious action in black-box settings. We evaluate AdapAM across eight multi-agent environments and compare it with four state-of-the-art and commonly-used baselines. Results demonstrate that AdapAM achieves the best attack performance in different perturbation rates. Besides, AdapAM-generated perturbations are the least noisy and hardest to detect, emphasizing the stealthiness.
BibTeX
@misc{chen2025adversarial,
  author = {Jianming Chen and Yawen Wang and Junjie Wang and Xiaofei Xie and Yuanzhe Hu and Qing Wang and Fanjiang Xu},
  title = {{Adversarial Attack on Black-Box Multi-Agent by Adaptive Perturbation}},
  year = {2025},
  eprint = {2511.15292},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.15292},
}
Figure