Evolution Under Fire: Evolving Agent Safety Harnesses with Adaptive Injection Stressors

Yujun Wang, Yilun Liu, Jinhe Bi, Krishna Kanth Nakka, Xuebing Zhou, Hinrich Sch{ü}tze, Yunpu MaarXiv 2026 · arXiv

TL;DR

Evolution Under Fire (EUF) is a framework that automatically designs safety defenses for LLM agents by co-evolving "harnesses" (the code surrounding the agent) with adaptive, targeted attacks. By pitting evolving defenses against increasingly sophisticated "stressors," the system discovers robust security mechanisms that significantly reduce prompt injection success while actually improving the agent's ability to complete its original tasks.

What Problem Does This Paper Address?

Language-model agents are increasingly being used to browse the web, read emails, and interact with external tools. This creates a massive security hole known as indirect prompt injection: an attacker can hide malicious instructions in a webpage or email that the agent then executes as if they were legitimate user commands.

The core challenge is the "same-channel" problem: legitimate task data and malicious instructions arrive through the same interface. Current defenses are mostly hand-designed, static, and suffer from a brutal security-utility trade-off. If you make a defense too strict, you break the agent's ability to do its job (utility); if you make it too permissive, it remains vulnerable (security). Because these defenses are manual, they can't keep up with the rapidly evolving landscape of adversarial attacks.

Methods

The authors propose a staged adversarial bootstrapping process called Evolution Under Fire. Instead of human engineers writing defense code, the system uses LLMs to evolve it through two interleaved processes:

  1. Safety Harness Evolution: The system treats the "harness"—the executable Python code that manages the agent's context, tool calls, and state—as a program to be evolved. It uses a utility-constrained Pareto tree search. A "Diagnoser" LLM analyzes execution failures, and an "Expander" LLM generates code edits. Crucially, it only keeps "utility-qualified" candidates—those that don't break the agent's core functionality.
  2. Adaptive Stressor Evolution: To prevent the defense from simply "overfitting" to a static benchmark, the system evolves stressors. These are adaptive attacks specifically designed to find residual weaknesses in the current best harnesses. These stressors must be "utility-preserving," meaning they shouldn't break the benign task itself, only trigger the malicious injection.
Figure 2: Illustration of EVOLUTION UNDER FIRE. Efficient harness evolution improves agent defenses, while adaptive stressor evolution exposes them to increasingly challenging attacks. Their feedback drives the search toward agents that remain capable and robust under adversarial pressure.

The framework uses a "Frozen Execution Layer" to ensure that the evolution process itself doesn't cheat by modifying the benchmark or the evaluation oracles.

Datasets

The researchers evaluated EUF on two primary benchmarks:

  • AgentDyn: A benchmark featuring open-ended tasks (like shopping or GitHub management) where legitimate completion often depends on external instructions, making it a realistic test for indirect injection.
  • Agent-SafetyBench: A broader benchmark covering various behavioral risks across interactive agent environments.

Key Insights and Ideas

  • Adversarial Bootstrapping is Key: Static benchmarks are a "fixed point" that defenses eventually exhaust. By evolving stressors alongside harnesses, the system creates a "moving target" that forces the discovery of more generalizable and robust defense strategies.
  • Utility as a Constraint, Not Just a Metric: By enforcing a hard utility floor (e.g., retaining at least 80% of the initial harness's success), the search avoids "collapsed" agents that are safe only because they are broken.
  • Branching vs. Linear Search: The authors found that a tree-structured search (allowing the system to revisit older, promising branches) is far more effective than a simple linear chain of edits. This prevents the evolution from getting stuck in local optima where a single bad edit ruins a lineage.
  • Transferability: Perhaps the most surprising finding is that harnesses evolved using one model (like GPT-4o-mini) transfer effectively to entirely different models (like Qwen3.5) without any further evolution. This suggests the evolved code discovers fundamental architectural patterns for safety.

Key Results

The results show that EUF occupies a superior "Pareto frontier" compared to existing hand-designed defenses like Spotlighting or DRIFT.

  • AgentDyn (GPT-4o-mini): The evolved harness slashed the Attack Success Rate (ASR) from 50.4% to just 3.0%, while actually increasing clean task utility from 45.0% to 46.7%.
  • Agent-SafetyBench (Qwen2.5-7B): ASR was reduced from 82.6% to 62.0%, outperforming the strongest reproduced baseline by 8.5 percentage points while maintaining 99.9% runtime validity.
  • Cross-Model Transfer: When the harness evolved for GPT-4o-mini was applied to Qwen3.5-35B-A3B, the ASR dropped from 32.1% to 0.9%, demonstrating that the evolved defenses are not model-specific.
Figure 1: Security–utility trade-off on AgentDyn. EVOLUTION UNDER FIRE combines near-maximal at- tack resistance with strong attacked-task utility, occupy- ing the favorable frontier across both objectives.

Limitations and Future Work

  • Compute Cost: The evolution process is computationally expensive, requiring hundreds of LLM calls for diagnosis, expansion, and evaluation.
  • Meta-Model Dependency: The quality of the evolved code depends on the "meta-model" (in this case, GPT-5) used to perform the evolution.
  • Fixed Environment: The current framework keeps the tools and environment fixed. Future work could explore evolving the tool definitions themselves to be more inherently secure.
  • Single Round of Feedback: The paper primarily explores one full round of stressor-guided feedback. Multiple iterative rounds could potentially push the frontier even further.

Summary updated

Abstract

Tool-using language-model agents must act on information from untrusted emails, webpages, and documents. This creates a central security challenge: malicious instructions and legitimate task directives can arrive through the same interaction channel. Existing defenses are largely hand-designed and static, making further improvements dependent on repeated manual redesign. We introduce Evolution Under Fire, a staged framework that couples safety-harness evolution with adaptive stressor evolution. Utility-constrained, Pareto-scheduled tree search evolves complete executable safety harnesses from execution evidence while keeping the task model, benchmark, tools, and evaluator fixed. Adaptive stressor evolution then discovers utility-preserving attacks that guide a subsequent stage of harness evolution. On AgentDyn with GPT-4o-mini, an evolved harness reduces attack success from 50.4% to 3.0%, improves clean and attacked utility over the initial harness, and offers a stronger security-utility trade-off than reproduced defenses. On Agent-SafetyBench with Qwen2.5-7B-Instruct, harness evolution reduces attack success from 82.6% to 62.0% while retaining helpfulness and runtime validity. Evolved harnesses also transfer without further evolution to Qwen3.5-35B-A3B, reducing attack success from 32.1% to 0.9% while improving attacked-task utility.
BibTeX
@misc{wang2026evolution,
  author = {Yujun Wang and Yilun Liu and Jinhe Bi and Krishna Kanth Nakka and Xuebing Zhou and Hinrich Sch\{ü\}tze and Yunpu Ma},
  title = {{Evolution Under Fire: Evolving Agent Safety Harnesses with Adaptive Injection Stressors}},
  year = {2026},
  eprint = {2611.22583},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22583},
}
Figure