Introduction
As multi-agent reinforcement learning (MARL) transitions from controlled research environments to real-world applications such as autonomous driving and industrial robotics, security has become a paramount concern. Most existing adversarial attack research operates under "white-box" assumptions, where the attacker has access to the model's internal weights, or assumes direct control over an agent's actions.
In practice, attackers typically face a black-box scenario, limited to observing agent inputs (observations) and outputs (actions). Furthermore, traditional attacks often target all agents or a fixed subset, making them either inefficient or easily detectable. The challenge is to develop an attack that is both effective—maximally reducing system rewards—and stealthy, utilizing minimal perturbations on a dynamically selected set of agents.
The AdapAM Framework
The authors introduce AdapAM, a framework designed to overcome black-box limitations through two primary modules:
- Adaptive Selection Policy: Instead of a blanket attack, AdapAM employs a hierarchical Reinforcement Learning (RL) policy based on Soft Actor-Critic (SAC). At each time step, it evaluates the environment state to determine the most "critical" victim agent and the specific "malicious action" that would inflict maximum damage.
- Proxy-based Perturbation: To generate effective perturbations without internal model access, the attacker trains "proxy agents" using Multi-Agent Generative Adversarial Imitation Learning (MAGAIL). These proxies mimic the target system's behavior. Once trained, a white-box C&W (Carlini & Wagner) attack is applied to the proxy to generate perturbations that successfully transfer to the black-box victim.
Experimental Setup
AdapAM was evaluated across eight environments from three prominent MARL benchmarks:
- StarCraft Multi-Agent Challenge (SMAC): Combat scenarios including
1c3s5z,8m, andbane_vs_bane. - Google Research Football (GRF): Cooperative tasks like
counterattackand3_vs_1. - Multi-Agent Particle Environments (MPE): Navigation and communication tasks such as
spreadandreference.
These benchmarks provide a diverse range of agent counts and task complexities to test the framework's generalizability.
Key Insights
- The "Critical Agent" Hypothesis: The study confirms that agent importance varies dynamically. By switching targets adaptively, AdapAM achieves superior damage with a lower perturbation budget.
- Imitation as a Bridge: Utilizing GAIL to create functional duplicates allows attackers to convert black-box constraints into white-box opportunities through observation.
- Stealth via Precision: AdapAM achieves stealthiness by perturbing only one agent at a time with minimal $L_\infty$ norm changes, making it difficult for standard anomaly detection to identify.
Key Results
AdapAM consistently outperformed state-of-the-art baselines like MASafe and AMCA across all tested environments. In several SMAC and Google Football scenarios, it reduced the victim's win rate from 100% to 0%.
The framework also demonstrated significant robustness against "Robust" MAS trained with adversarial defenses. Furthermore, in tests against anomaly detection algorithms, AdapAM's perturbations yielded the lowest F1 scores for detection, confirming its high stealthiness.
Limitations and Future Work
Despite its effectiveness, AdapAM requires substantial observation data to train proxy agents via imitation learning. Additionally, scaling the "Adaptive Selection Policy" for massive-scale swarms remains a challenge. Future research will focus on leveraging these "red teaming" insights to develop more sophisticated, adaptive defense mechanisms capable of mitigating such surgical adversarial strikes.
Summary updated