Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systems

Pierre Peigné, Mikolaj Kniejski, Filip Sondej, Matthieu David, Jason Hoelscher-Obermaier, Christian Schröder de Witt, Esben KranAAAI 2025 · DOI

Abstract

As Large Language Model (LLM) applications evolve from single-turn interactions to autonomous Multi-Agent Systems (MAS), they face new security challenges. This paper investigates "multi-hop prompt injection," where malicious instructions spread through agent networks like digital worms. We introduce the concept of the "Security Tax"—the performance penalty in agent cooperation and helpfulness incurred when implementing security defenses. Through simulations in a chemical research environment, we evaluate various defense mechanisms, finding that "Memory Vaccines" offer a superior balance between system robustness and agent utility compared to traditional instruction-based defenses.

Introduction: The Vulnerability of Collaboration

Multi-Agent Systems (MAS) leverage specialized agents communicating via natural language to solve complex, multi-step goals. However, the inherent helpfulness and connectivity of these agents create a significant security vulnerability. An attacker compromising a single agent can inject "infectious prompts" that propagate across the network. Because agents are designed to be cooperative, they may unwittingly execute and spread malicious instructions, potentially leading to catastrophic outcomes in physical or digital environments.

Methodology

The researchers utilized a simulated autonomous chemical research facility based on the "ChemCrow" environment, consisting of seven specialized agents.

Attack Scenarios

The system was subjected to 12 unique "jailbreak" prompts designed to manipulate the agents into performing dangerous chemical reactions, such as causing a lab explosion.

Defense Mechanisms

Four types of interventions were tested:

  1. Safety Instructions: Generic safety rules added to the system prompt.
  2. Passive: Instructions to refuse malicious input.
  3. Active: Instructions to refuse and proactively warn other agents.
  4. Memory Vaccines: Injecting "fake memories" into an agent's history demonstrating the successful handling of a previous malicious request.
  5. Passive: Shows refusal of a bad request.
  6. Active: Shows refusal and alerting the team.

Evaluation Metrics

The study measured two primary metrics:

  • System Robustness: The percentage of simulations where the malicious goal (e.g., the explosion) was successfully prevented.
  • Agent Cooperation: The compliance rate of agents when given "harmless but unusual" instructions (e.g., urgent resource redirection). This metric quantifies the "Security Tax."

Key Results and the "Security Tax"

The evaluation across multiple models (GPT-3.5-turbo, GPT-4-turbo, GPT-4o, and GPT-4o-mini) revealed a clear trade-off between security and utility.

System robustness vs. agent cooperation across different models and defense strategies. Memory vaccines (blue) generally maintain higher cooperation than instruction-based defenses (green).

The Efficiency of Memory Vaccines

Memory Vaccines proved significantly more effective than instruction-based defenses. While instruction-based defenses often made agents overly rigid and suspicious, vaccines provided a behavioral template that maintained higher cooperation. On average, Active Vaccines increased system robustness from 76.7% to 90.0% while maintaining a cooperation rate of approximately 87%. In contrast, instruction-based defenses caused cooperation to plummet; for GPT-4o-2024-08-06, cooperation dropped from 91.7% to just 16.7% with active instructions.

Model-Specific Vulnerabilities

Larger models like GPT-4o and GPT-4-turbo exhibited higher baseline safety and reached 100% robustness with certain defenses. However, smaller models like GPT-4o-mini were notably more fragile, showing only 50% robustness without intervention and remaining more exploitable even with defenses.

Discussion and Limitations

The "Security Tax" highlights a fundamental challenge in MAS design: making agents safe without making them useless. Active defenses, while powerful at containing "worms," can trigger false alarms that disrupt legitimate operations.

Future research is needed to explore:

  • Domain Generalization: Testing these dynamics in non-chemical environments like software engineering.
  • Adversarial Sophistication: Evaluating defenses against more advanced automated prompt injection techniques.
  • Scalability: Understanding how infectious prompts behave in larger networks of hundreds or thousands of agents.

Conclusion

Securing Multi-Agent Systems requires moving beyond simple safety prompts. Memory Vaccines represent a promising, "surgical" approach to defense that preserves the collaborative utility of AI agents while providing robust protection against infectious prompt injections.

Summary updated

BibTeX
@inproceedings{peigne2025multi,
  author = {Pierre Peigné and Mikolaj Kniejski and Filip Sondej and Matthieu David and Jason Hoelscher-Obermaier and Christian Schröder de Witt and Esben Kran},
  title = {{Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systems}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i26.34970},
}
Figure