Poisoning the Well: Can We Simultaneously Attack a Group of Learning Agents?

Ridhima Bector, Hang Xu, Abhay Aradhya, Chai Quek, Zinovi RabinovichIJCAI 2023 · DOI

Introduction

As reinforcement learning (RL) agents transition from research labs to real-world deployments—such as drone swarms or autonomous warehouse fleets—the security of these collective systems becomes paramount. While adversarial RL has extensively studied attacks on individual agents, the challenge of attacking a population of learners remains largely unexplored. This paper introduces the Collective Environment Poisoning (CEP) framework, the first method designed to simultaneously manipulate a group of RL agents by subtly altering their shared environment.

The Collective Environment Poisoning (CEP) Framework

The core of the CEP framework is a bi-level hierarchical structure where an Attacker Markov Decision Process (MDP) operates above a Victim Population MDP. Unlike traditional attacks that require internal access to an agent's weights or rewards, CEP operates in an Ultra-Blackbox (UB) setting. The attacker only observes the agents' movement traces and lacks any knowledge of their internal states or reward signals.

The Collective Environment Poisoning (CEP) framework architecture, highlighting the interaction between the attacker, the environment, and the victim population.

To overcome individual differences and scalability issues, the framework employs two key innovations:

  1. Behavioral Embedding: A Variational Auto-Encoder (VAE) maps raw observation traces into a latent representation of each agent's policy.
  2. Wasserstein Barycenters: To represent the entire population regardless of size, the attacker calculates the Wasserstein barycenter of the individual VAE embeddings. This provides a fixed-size, geometric summary of the group's collective behavior.

Methodology and Scenarios

The researchers tested CEP in a 3D Grid World where navigational agents seek the shortest path. The attacker, acting as an "elevation expert," modifies grid cell altitudes to change transition probabilities. This "poisoning" steers agents toward a non-optimal target path. The study evaluates three collective settings: Implicit (independent agents), Swarm (anonymous interaction), and True (social learning).

Experimental Results and Key Insights

The experiments revealed that the barycenter-based approach significantly outperforms standard data concatenation, especially as population size increases. Concatenation often introduces noise that confuses the attacker, whereas the Wasserstein barycenter provides a clean, actionable summary.

Performance metrics comparing Barycenter and Concatenation methods across different population sizes. The plots show Attack Accuracy, Softmax Accuracy, and Attacker Effort over time.

Key findings include:

  • High Adoption Rates: In implicit collective scenarios, the attack achieved near 100% accuracy, forcing the entire population to adopt the target behavior.
  • The "Sawtooth" Learning Curve: As seen in the accuracy plots, performance often follows a sawtooth pattern. Each environment modification initially drops agent performance, but as they adapt to the "poisoned" well, they converge more tightly onto the attacker's desired path.
  • Size-Agnostic Transfer: Attacks trained on a small group (e.g., 3 agents) successfully transferred to much larger populations (e.g., 20 agents) without retraining, demonstrating the robustness of the barycenter representation.

Conclusion and Future Work

The CEP framework demonstrates that collective systems are vulnerable to environment-based poisoning even under strict blackbox constraints. Future research will focus on extending this method to continuous state-action spaces, heterogeneous populations with diverse goals, and "open" systems where agent membership changes dynamically.

Summary updated

BibTeX
@inproceedings{bector2023poisoning,
  author = {Ridhima Bector and Hang Xu and Abhay Aradhya and Chai Quek and Zinovi Rabinovich},
  title = {{Poisoning the Well: Can We Simultaneously Attack a Group of Learning Agents?}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/386},
}
Figure