TL;DR
This paper introduces an automated, pre-deployment pipeline that scans, hardens, and validates LLM-based multi-agent applications to prevent data leakage and tool misuse. By applying targeted code patches and prompt modifications before deployment, the pipeline eliminates data leaks from basic jailbreaks entirely and reduces leaks from complex, stress-induced attacks by 91%—all without sacrificing the application's core utility or requiring heavy runtime monitoring.
What Problem Does This Paper Address?
As large language model (LLM) agents transition from simple chatbots to production-grade systems, they are increasingly trusted with external tools (like databases, emails, and file systems) and private data. Frameworks like LangGraph and CrewAI allow multiple agents to collaborate, passing data and instructions back and forth.
However, this flexibility creates a massive attack surface. If an agent ingests untrusted data (e.g., a malicious email or a poisoned web page), an attacker can execute an indirect prompt injection. This can trick the agent into overriding its system instructions, misusing its tools, or leaking sensitive information (like credentials or PII) across agent boundaries.
Current defenses usually rely on runtime monitoring or fragile prompt engineering. Runtime defenses add significant computational overhead and fail to fix the underlying code vulnerabilities, while prompt engineering often breaks when faced with novel inputs. The authors address this gap by treating data leakage as an architectural, design-time flaw that can be fixed before the code is ever deployed.
Methods
The authors propose a continuous integration/continuous delivery (CI/CD) oriented pipeline consisting of three sequential modules:
- Analyzer Module: This module performs static analysis using Python Abstract Syntax Trees (ASTs) to map out the application's topology (agents, tools, memory access, and communication paths). It then uses an LLM with Chain-of-Thought (CoT) reasoning to trace how data flows through the system, identifying and ranking potential leakage paths based on risk severity.
- Modifier Module: Instead of globally locking down the system, the modifier applies localized, minimally invasive patches to the highest-risk paths. It rewrites prompt templates to strictly separate data from instructions using delimiters, strips unsafe context from memory retrievals, enforces role separation between agents, and tightens tool schemas (e.g., converting free-form string arguments into strict allowlists).
- Validation Module: To ensure the patches actually work without breaking the application, the pipeline runs an automated, iterative testing loop. An LLM-based "red team" generates adversarial inputs (jailbreaks, instruction overrides, stress-induced prompts), while a separate branch tests benign inputs to verify that the application's original functionality (utility) remains intact.
Datasets
The pipeline was evaluated on two distinct sets of targets:
- Five Real-World Case Studies: Custom and adapted multi-agent applications built using CrewAI and LangGraph. These included a Network Monitoring Assistant (operating over an O-RAN telecom database), an HR Assistant, an Automated Trip Planner, an Automated Email Responder, and an MCP-based Candidate Hiring application.
- The AgentDojo Benchmark: A standardized benchmark featuring 97 realistic user tasks, 629 security test cases, and 70 tools spread across four domains (Workspace, Slack, Banking, and Travel).
Key Insights and Ideas
- Data Leakage is a Build-Time Problem: The core philosophy of this paper is that data leakage often stems from recurring engineering anti-patterns (like unsafe I/O handling or over-privileged tool bindings). Fixing these at the code level before deployment is more robust than trying to police them at runtime.
- The Security-Utility Tradeoff: A major pitfall of many security tools is "over-hardening"—blocking so many actions that the application becomes useless. The authors show that by using an iterative validation loop (detailed in Figure 2), the system can fine-tune patches so they block attacks but let legitimate traffic pass.
- Context Matters for Risk Ranking: Structural code analysis alone cannot determine if a data flow is dangerous. Combining static AST parsing with semantic LLM reasoning allows the pipeline to understand what the data actually represents (e.g., identifying that a specific string variable holds an InfluxDB password).
Key Results
- 100% Leakage Elimination on Basic Attacks: Across all five case-study applications, the hardened code completely eliminated data leaks caused by direct requests, basic jailbreaks, and instruction-override attacks.
- 91% Reduction in Stress-Induced Attacks: Under highly coercive, urgency-driven prompting (the hardest category to defend against), the pipeline reduced the Attack Success Rate (ASR) from an average of 81.9% down to just 7.4%.
- AgentDojo Performance: When tested against the standardized benchmark, the pipeline improved Benign Utility Under Attack by 35.2 percentage points compared to AgentDojo's default defense, while achieving a net security gain of +7.4 points. Instead of completely shutting down when a suspicious input was detected, the hardened agents safely completed their tasks.
Limitations and Future Work
While highly effective, the authors note a few remaining challenges:
- Residual Stress Vulnerabilities: In complex environments like the Network Monitoring Assistant, clever, stress-framed prompts still achieved a 7.4% success rate.
- Schema-Preserving Tampering: If an attacker manipulates data flowing between agents in a way that perfectly matches the expected structural schema, the pipeline's static checks can be bypassed.
- Multi-User Isolation: The current framework does not inherently enforce strict multi-tenant authorization boundaries.
Future work will focus on developing data provenance tracking for inter-agent communication, implementing tighter memory scoping, and integrating this pre-deployment pipeline with runtime information-flow control (IFC) for a defense-in-depth approach.
Summary updated