Locks Tested Without Burglars: Using Coding Assistants to Break Prompt Injection Defenses

Atharv Singh Patlan, Pramod Viswanath, Prateek MittalReliable ML @ NeurIPS 2025 · OpenReview

TL;DR

This paper argues that current evaluations of prompt injection defenses for LLMs are too simplistic, using "toy" benchmarks that don't reflect real-world adversaries. The authors demonstrate that AI coding assistants can act as powerful automated red-teamers, effectively reverse-engineering defense code and crafting adaptive attacks that significantly increase success rates, highlighting a critical need for more robust evaluation methodologies.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly integrated into applications, but they face a significant security vulnerability known as "prompt injection." This occurs when malicious users craft inputs that manipulate the LLM into performing unintended actions, bypassing its intended functionality. Numerous defenses have been proposed to combat this, ranging from input sanitization to more complex detection mechanisms. However, the authors point out a critical flaw in how these defenses are evaluated: they are typically tested against small, static datasets of hand-crafted "attacks." This approach fails to capture the adaptive and sophisticated nature of real-world adversaries who can probe systems, understand their logic, and craft dynamic attacks. The paper aims to bridge this gap by proposing and evaluating a new paradigm for testing these defenses.

Methods

The core contribution of this paper is the proposal and demonstration of using AI coding assistants as automated red-teamers for prompt injection defenses. The authors developed a structured prompting strategy for an AI assistant (specifically Claude Code, using Claude-Sonnet-4) that guides it through four phases:

  1. Extraction: The assistant is tasked with identifying and extracting the literal prompts, required return formats, and any hard-coded assumptions within the defense's codebase.
  2. Audit: The assistant analyzes the extracted information to summarize the defense's architecture and pinpoint potential weaknesses.
  3. Attack Generation: Based on the extracted information, the assistant constructs an adaptive attack wrapper. This wrapper aims to mimic the defense's expected format, embed a malicious payload, and subvert the defense's logic while forcing the LLM to perform the injected task.
  4. Iterative Refinement: The assistant is allowed to refine its attack up to four times to successfully break the defense.

This pipeline was applied to three state-of-the-art defenses: DataSentinel, MELON, and DRIFT.

Datasets

The paper evaluates the defenses across three types of benchmarks:

  • Standard task benchmarks: These are small, task-specific datasets used in the original papers introducing the defenses.
  • AgentDojo [2]: A standardized benchmark with a broader range of tasks and security test cases, specifically the Slack environment was used for MELON.
  • CrAIBench [5] (argument-perturbation setting): This benchmark was used specifically for DRIFT, focusing on attacks that manipulate tool arguments rather than substituting tools entirely. This is highlighted as a more realistic adversary model.

The paper does not introduce new datasets but rather uses existing ones to evaluate the effectiveness of assistant-generated attacks against them.

Key Insights and Ideas

  • AI Coding Assistants as Adversarial Collaborators: The most significant insight is that AI coding assistants are not just productivity tools but can act as sophisticated, automated adversaries. Their ability to parse code, understand logic, and generate novel inputs makes them powerful red-teaming partners.
  • The "Toy Burglar" vs. "Master Locksmith" Gap: The paper vividly illustrates that current benchmarks test defenses against simple, predictable attacks (toy burglars), while real adversaries are more like master locksmiths who can adapt, disguise, and dismantle defenses.
  • Adaptive Attacks are Crucial: The research strongly suggests that static, non-adaptive attacks are insufficient for evaluating prompt injection defenses. Adaptive attacks, informed by knowledge of the defense's inner workings, are far more effective and reveal vulnerabilities missed by simpler testing.
  • Defense Code Reveals Vulnerabilities: The ability of coding assistants to extract detailed prompts and architectural assumptions directly from defense code is a key enabler of their success. This highlights the importance of secure coding practices even within defense mechanisms.

Key Results

  • Extraction Fidelity: The AI assistant successfully extracted key prompts, required formats, and architectural assumptions from all three defenses tested (DRIFT, DataSentinel, MELON). While mostly accurate, multi-prompt defenses like DRIFT showed some missed components.
  • Attack Success Rate (ASR) Increase: Adaptive attacks generated by the AI assistant dramatically outperformed baseline non-adaptive attacks across all defenses:
  • DataSentinel: ASR increased from 0.00% to 62%.
  • MELON: ASR increased from 0.80% to 33.3%.
  • DRIFT: ASR increased from 1.3% to 50%.
  • These results demonstrate that coding assistants can craft attacks that are significantly more effective, raising attack success rates by up to 50-60% in some cases.

Limitations and Future Work

  • Cross-Assistant Comparison: The paper acknowledges that its evaluation focused on a single coding assistant (Claude Code). Future work should compare the red-teaming capabilities of other assistants like Codex, Cursor, and Gemini-CLI to assess the generality of these findings.
  • "Final Attack" vs. Evaluation Paradigm: The authors emphasize that their goal is not to claim a "final attack" but to advocate for a new evaluation paradigm.
  • Recommendation for Defense Authors: The paper strongly recommends that authors proposing new prompt injection defenses should evaluate them against assistant-driven red-teaming before publication. This aligns with the broader trend in ML security where adversarial evaluation is becoming standard practice.
  • Building Robust Red-Teaming Pipelines: The authors hope their work will spur further research into building integrated adversarial pipelines that make this type of testing a default part of the defense evaluation process.

Summary updated

Abstract

Prompt injection is a critical security challenge for large language models (LLMs), yet proposed defenses are typically evaluated on toy benchmarks that fail to reflect real adversaries. We show that AI coding assistants, such as Claude Code, can act as automated red-teamers: they parse defense code, uncover hidden prompts and assumptions, and generate adaptive natural-language attacks. Evaluating three recent defenses -- DataSentinel, Melon, and DRIFT -- across standard and realistic benchmarks, we find that assistants extract defense logic and craft attacks that raise attack success rates by up to 50–60\%. These results suggest coding assistants are not just productivity tools but practical adversarial collaborators, and that defenses should be tested against them before claims of robustness are made.
BibTeX
@inproceedings{patlan2025locks,
  author = {Atharv Singh Patlan and Pramod Viswanath and Prateek Mittal},
  title = {{Locks Tested Without Burglars: Using Coding Assistants to Break Prompt Injection Defenses}},
  booktitle = {Reliable ML @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=ER5Qmhfd5i},
}
Figure