Scaling Adaptive Prompt Injection Attacks with Agent Swarms

Alexey Kurakin, Vishal Karande, Alex Bie, Chongyang Shi, Shuang Song, Saeed Mahloujifar, Abdullatif Köksal, Jamie Hayes, Juliette Pluto, Yinghao Sun, Kamalika ChaudhuriarXiv 2026 · arXiv

TL;DR

Despite the apparent "saturation" of current prompt injection benchmarks, this paper demonstrates that frontier AI models remain highly vulnerable to sophisticated, adaptive attacks. By deploying a collaborative "Agentic Optimizer" swarm, the authors successfully bypassed defenses in top-tier models like Gemini 4 Argon, proving that prompt injection is far from a solved problem.

What Problem Does This Paper Address?

Prompt injection occurs when a third party inserts malicious instructions into data processed by an AI agent (e.g., a hidden command in a PDF that tells an email assistant to delete messages). While early, static attacks were largely mitigated by mid-2025, a new "plateau" has emerged where human red-teaming and existing benchmarks suggest models are becoming robust.

The authors argue this robustness is an illusion caused by two factors:

  1. Benchmark Saturation: Existing tests use static payloads that modern models have learned to recognize.
  2. Conflating Safety with Robustness: Many benchmarks use explicitly "evil" goals. Models might block these because they recognize "malicious intent" rather than because they are actually following the correct instruction hierarchy.

The paper seeks to determine if frontier models can still be subverted by highly capable, automated attackers using non-malicious but "orthogonal" goals to test pure architectural robustness.

Methods

The core technical contribution is the Agentic Optimizer, a swarm-based framework for generating adaptive prompt injections. Unlike previous methods that rely on a single LLM to iterate on a payload, this approach uses:

  • Agent Swarms: $N$ teams of $M$ agents working in parallel.
  • Collaborative Tools: Agents use a shared "whiteboard" to exchange notes and "leaderboards" to track the most successful injection candidates found by the swarm.
  • Optimization Strategies: A database of natural language strategies (e.g., "make the target call look like a data dependency") that agents can follow and refine.
  • Black-box Feedback: The optimizer only requires the score of an attack to iterate, making it effective even against models where weights or internal states are hidden.

Datasets

The authors introduced a new synthetic evaluation dataset of 100 tool-use scenarios.

  • Design: Each scenario features a "task domain" (e.g., tutoring) and an unrelated "target domain" (e.g., health tracking).
  • The Goal: The attacker’s goal is to trick the model into calling a tool from the target domain instead of completing the user's task.
  • Why it's different: By using benign actions (like logging heart rate) as the adversarial goal, the dataset disentangles a model's security knowledge (knowing that "delete all files" is bad) from its instruction hierarchy (knowing that tool output should not override user instructions).
Figure 1: Example scenario from our dataset. Tools from both the task domain (tutoring) and the target domain (health_tracking) are made available to the model. The model is prompted with the user goal, which results in a trajectory making task domain tool calls. The injection is inserted in the returned result for the final tool call (schedule_tutoring_session). While a regular return results in the model responding with a booking summary, a poisoned return hijacks the next model turn into calling log_vital_signs, thus fulfilling the attacker goal.

Key Insights and Ideas

  • Attacker Capability Scales with Model Strength: One of the most significant findings is that more capable models (like Gemini 4 Argon) actually make better attackers. They are more adept at rephrasing injections to avoid suspicion and coordinating within the swarm to find exploits.
  • Instruction Hierarchy is the Weak Link: Even when models are "safe," they struggle to prioritize the user's original prompt over instructions found within data (tool outputs).
  • The "Move Second" Advantage: Adaptive attackers who can see how a model responds to an initial attempt and then pivot their strategy are significantly more successful than static attacks.
  • Chain-of-Thought (CoT) as a Double-Edged Sword: While CoT helps models reason, if the attacker can see the defender's CoT, the attack success rate increases, as the attacker can "see" why the previous attempt failed.

Key Results

The results show a stark reality: no model tested was fully robust.

  • Frontier Vulnerability: Even the most robust model, Gemini 4 Argon, only resisted 66% of attacks from the strongest version of the Agentic Optimizer. When the attacker's query budget was limited to 150, the resistance rate (ARR) rose to 87%, but still showed significant failure.
  • Open-Weights Gap: Popular open-weights models like Kimi K3 and DeepSeek-V4.1-Flash showed nearly zero resistance, with ARRs of 1.0%, despite performing well on standard capability benchmarks.
Table 1: ARR (%) of defenders against the strongest attacker (Gemini 4 Argon), defender thinking on, CoT hidden. Lower means a stronger attack.

The study also confirmed that as the "intelligence" of the attacker model increases, the resistance of the defender model consistently drops across the board.

Table 2: ARR@150 (%) for different pairs of attacker and defender, lower ARR means a stronger attack. Defender operates with high level of thinking, but CoT are invisible to attacker.

Limitations and Future Work

  • Reproducibility: Most high-performing adaptive attackers (including the one in this paper) rely on closed-source frontier models, making it difficult for the broader research community to reproduce these results without significant API costs.
  • Computational Budget: Running a swarm of agents for every evaluation is computationally expensive compared to static benchmarking.
  • Defense-in-Depth: The authors conclude that "model hardening" (training the model to be better) is likely insufficient on its own. Future work should focus on "full stack" defenses, potentially involving architectural changes that strictly separate data from instructions.

Summary updated

Abstract

Prompt injections have emerged as one of the top security risks faced by AI agents today. Early work in prompt injection attacks looked at simple tool-call agents and static attack payloads, and were saturated since mid-2025. As of Fall 2026, human red-teaming may be nearing saturation, especially on the Gemini family of models. In this work, we argue that despite this apparent plateau, prompt injection attacks are still possible and continued effort into prompt injection defense is needed. We design a new synthetic evaluation dataset which uses random non-malicious actions as adversarial goals. Such dataset design helps us to evaluate prompt injection robustness as a model ability to respect instruction hierarchy, and disentangle this ability from model knowledge of security practices. We show that an agentic attacker of increased capabilities can reliably trick model to follow instructions in tool output in our evaluation, even for models previously considered robust on public prompt injection benchmarks such as GraySwan Arena. This shows that current frontier defenses may require proactive, automated red-teaming to remain resilient to increasingly capable adversaries.
BibTeX
@misc{kurakin2026scaling,
  author = {Alexey Kurakin and Vishal Karande and Alex Bie and Chongyang Shi and Shuang Song and Saeed Mahloujifar and Abdullatif Köksal and Jamie Hayes and Juliette Pluto and Yinghao Sun and Kamalika Chaudhuri},
  title = {{Scaling Adaptive Prompt Injection Attacks with Agent Swarms}},
  year = {2026},
  eprint = {2611.11861},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11861},
}
Figure