Data Leakage Prevention in Agentic Applications via Preemptive Hardening

Akansha Shukla, Emily Bellov, Parth Atulbhai Gandhi, Yuval Elovici, Asaf ShabtaiarXiv 2026 · arXiv

TL;DR

This paper introduces an automated, pre-deployment pipeline that scans, hardens, and validates LLM-based multi-agent applications to prevent data leakage and tool misuse. By applying targeted code patches and prompt modifications before deployment, the pipeline eliminates data leaks from basic jailbreaks entirely and reduces leaks from complex, stress-induced attacks by 91%—all without sacrificing the application's core utility or requiring heavy runtime monitoring.

What Problem Does This Paper Address?

As large language model (LLM) agents transition from simple chatbots to production-grade systems, they are increasingly trusted with external tools (like databases, emails, and file systems) and private data. Frameworks like LangGraph and CrewAI allow multiple agents to collaborate, passing data and instructions back and forth.

However, this flexibility creates a massive attack surface. If an agent ingests untrusted data (e.g., a malicious email or a poisoned web page), an attacker can execute an indirect prompt injection. This can trick the agent into overriding its system instructions, misusing its tools, or leaking sensitive information (like credentials or PII) across agent boundaries.

Current defenses usually rely on runtime monitoring or fragile prompt engineering. Runtime defenses add significant computational overhead and fail to fix the underlying code vulnerabilities, while prompt engineering often breaks when faced with novel inputs. The authors address this gap by treating data leakage as an architectural, design-time flaw that can be fixed before the code is ever deployed.

Methods

The authors propose a continuous integration/continuous delivery (CI/CD) oriented pipeline consisting of three sequential modules:

  1. Analyzer Module: This module performs static analysis using Python Abstract Syntax Trees (ASTs) to map out the application's topology (agents, tools, memory access, and communication paths). It then uses an LLM with Chain-of-Thought (CoT) reasoning to trace how data flows through the system, identifying and ranking potential leakage paths based on risk severity.
  2. Modifier Module: Instead of globally locking down the system, the modifier applies localized, minimally invasive patches to the highest-risk paths. It rewrites prompt templates to strictly separate data from instructions using delimiters, strips unsafe context from memory retrievals, enforces role separation between agents, and tightens tool schemas (e.g., converting free-form string arguments into strict allowlists).
  3. Validation Module: To ensure the patches actually work without breaking the application, the pipeline runs an automated, iterative testing loop. An LLM-based "red team" generates adversarial inputs (jailbreaks, instruction overrides, stress-induced prompts), while a separate branch tests benign inputs to verify that the application's original functionality (utility) remains intact.
Fig. 1. Overview of the preemptive hardening pipeline for data leakage prevention in AI-agent applications. The analyzer module (left) discovers agents, maps tools, and inspects data-flow paths to produce a ranked audit report. The modifier module (top) applies targeted transformations to mitigate identified risks. The validation module (bottom) verifies benign operation preservation and security enforcement on the hardened application.

Datasets

The pipeline was evaluated on two distinct sets of targets:

  • Five Real-World Case Studies: Custom and adapted multi-agent applications built using CrewAI and LangGraph. These included a Network Monitoring Assistant (operating over an O-RAN telecom database), an HR Assistant, an Automated Trip Planner, an Automated Email Responder, and an MCP-based Candidate Hiring application.
  • The AgentDojo Benchmark: A standardized benchmark featuring 97 realistic user tasks, 629 security test cases, and 70 tools spread across four domains (Workspace, Slack, Banking, and Travel).

Key Insights and Ideas

  • Data Leakage is a Build-Time Problem: The core philosophy of this paper is that data leakage often stems from recurring engineering anti-patterns (like unsafe I/O handling or over-privileged tool bindings). Fixing these at the code level before deployment is more robust than trying to police them at runtime.
  • The Security-Utility Tradeoff: A major pitfall of many security tools is "over-hardening"—blocking so many actions that the application becomes useless. The authors show that by using an iterative validation loop (detailed in Figure 2), the system can fine-tune patches so they block attacks but let legitimate traffic pass.
  • Context Matters for Risk Ranking: Structural code analysis alone cannot determine if a data flow is dangerous. Combining static AST parsing with semantic LLM reasoning allows the pipeline to understand what the data actually represents (e.g., identifying that a specific string variable holds an InfluxDB password).
Fig. 2. Validation module overview.

Key Results

  • 100% Leakage Elimination on Basic Attacks: Across all five case-study applications, the hardened code completely eliminated data leaks caused by direct requests, basic jailbreaks, and instruction-override attacks.
  • 91% Reduction in Stress-Induced Attacks: Under highly coercive, urgency-driven prompting (the hardest category to defend against), the pipeline reduced the Attack Success Rate (ASR) from an average of 81.9% down to just 7.4%.
  • AgentDojo Performance: When tested against the standardized benchmark, the pipeline improved Benign Utility Under Attack by 35.2 percentage points compared to AgentDojo's default defense, while achieving a net security gain of +7.4 points. Instead of completely shutting down when a suspicious input was detected, the hardened agents safely completed their tasks.

Limitations and Future Work

While highly effective, the authors note a few remaining challenges:

  • Residual Stress Vulnerabilities: In complex environments like the Network Monitoring Assistant, clever, stress-framed prompts still achieved a 7.4% success rate.
  • Schema-Preserving Tampering: If an attacker manipulates data flowing between agents in a way that perfectly matches the expected structural schema, the pipeline's static checks can be bypassed.
  • Multi-User Isolation: The current framework does not inherently enforce strict multi-tenant authorization boundaries.

Future work will focus on developing data provenance tracking for inter-agent communication, implementing tighter memory scoping, and integrating this pre-deployment pipeline with runtime information-flow control (IFC) for a defense-in-depth approach.

Summary updated

Abstract

Agentic systems integrate LLM driven planning with interfaces to external tools, making data leakage and tool misuse feasible via instruction/data boundary failures and prompt injection attacks. Enforcing required controls consistently is particularly challenging in workflows spanning many codebases and heterogeneous agents. To address this challenge in multi agentic systems, we present a pre-deployment pipeline for scanning, hardening, and validation of agentic applications. The pipeline analyzes prompt templates, tool interfaces, and tool-invocation code to identify leakage-enabling patterns and generate actionable patches. The hardened application is then validated through adversarial prompt injection attacks and benign input variations ensuring that mitigations do not disrupt intended behavior. In the hardening stage, high-risk tools are prioritized, and minimally invasive mitigations are applied, including schema tightening, boundary sanitization, allowlist-based tool gating, and least-privilege checks. In the validation stage, the pipeline automatically generates attack inputs that mimic jailbreaks, instruction overrides, and tool-targeted manipulation, along with benign task variants, to confirm that the functionality of the hardened application is preserved after remediation. We evaluated the pipeline on five real-world agentic applications, as well as on the AgentDojo benchmark. Across all applications, the proposed pipeline identified recurring leakage-enabling patterns and generated patches that can be integrated without disrupting the intended application behavior. The resulting modifications of application code were shown to eliminate leaks when targeted by basic jailbreak and instruction-override attacks, achieving a 100% reduction in leakage, and reduce leaks by 91% under conditions of stress-induced manipulation, without the need of continuous runtime policy enforcement.
BibTeX
@misc{shukla2026data,
  author = {Akansha Shukla and Emily Bellov and Parth Atulbhai Gandhi and Yuval Elovici and Asaf Shabtai},
  title = {{Data Leakage Prevention in Agentic Applications via Preemptive Hardening}},
  year = {2026},
  eprint = {2607.18847},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.18847},
}
Figure