TL;DR
Evolution Under Fire (EUF) is a framework that automatically designs safety defenses for LLM agents by co-evolving "harnesses" (the code surrounding the agent) with adaptive, targeted attacks. By pitting evolving defenses against increasingly sophisticated "stressors," the system discovers robust security mechanisms that significantly reduce prompt injection success while actually improving the agent's ability to complete its original tasks.
What Problem Does This Paper Address?
Language-model agents are increasingly being used to browse the web, read emails, and interact with external tools. This creates a massive security hole known as indirect prompt injection: an attacker can hide malicious instructions in a webpage or email that the agent then executes as if they were legitimate user commands.
The core challenge is the "same-channel" problem: legitimate task data and malicious instructions arrive through the same interface. Current defenses are mostly hand-designed, static, and suffer from a brutal security-utility trade-off. If you make a defense too strict, you break the agent's ability to do its job (utility); if you make it too permissive, it remains vulnerable (security). Because these defenses are manual, they can't keep up with the rapidly evolving landscape of adversarial attacks.
Methods
The authors propose a staged adversarial bootstrapping process called Evolution Under Fire. Instead of human engineers writing defense code, the system uses LLMs to evolve it through two interleaved processes:
- Safety Harness Evolution: The system treats the "harness"—the executable Python code that manages the agent's context, tool calls, and state—as a program to be evolved. It uses a utility-constrained Pareto tree search. A "Diagnoser" LLM analyzes execution failures, and an "Expander" LLM generates code edits. Crucially, it only keeps "utility-qualified" candidates—those that don't break the agent's core functionality.
- Adaptive Stressor Evolution: To prevent the defense from simply "overfitting" to a static benchmark, the system evolves stressors. These are adaptive attacks specifically designed to find residual weaknesses in the current best harnesses. These stressors must be "utility-preserving," meaning they shouldn't break the benign task itself, only trigger the malicious injection.
The framework uses a "Frozen Execution Layer" to ensure that the evolution process itself doesn't cheat by modifying the benchmark or the evaluation oracles.
Datasets
The researchers evaluated EUF on two primary benchmarks:
- AgentDyn: A benchmark featuring open-ended tasks (like shopping or GitHub management) where legitimate completion often depends on external instructions, making it a realistic test for indirect injection.
- Agent-SafetyBench: A broader benchmark covering various behavioral risks across interactive agent environments.
Key Insights and Ideas
- Adversarial Bootstrapping is Key: Static benchmarks are a "fixed point" that defenses eventually exhaust. By evolving stressors alongside harnesses, the system creates a "moving target" that forces the discovery of more generalizable and robust defense strategies.
- Utility as a Constraint, Not Just a Metric: By enforcing a hard utility floor (e.g., retaining at least 80% of the initial harness's success), the search avoids "collapsed" agents that are safe only because they are broken.
- Branching vs. Linear Search: The authors found that a tree-structured search (allowing the system to revisit older, promising branches) is far more effective than a simple linear chain of edits. This prevents the evolution from getting stuck in local optima where a single bad edit ruins a lineage.
- Transferability: Perhaps the most surprising finding is that harnesses evolved using one model (like GPT-4o-mini) transfer effectively to entirely different models (like Qwen3.5) without any further evolution. This suggests the evolved code discovers fundamental architectural patterns for safety.
Key Results
The results show that EUF occupies a superior "Pareto frontier" compared to existing hand-designed defenses like Spotlighting or DRIFT.
- AgentDyn (GPT-4o-mini): The evolved harness slashed the Attack Success Rate (ASR) from 50.4% to just 3.0%, while actually increasing clean task utility from 45.0% to 46.7%.
- Agent-SafetyBench (Qwen2.5-7B): ASR was reduced from 82.6% to 62.0%, outperforming the strongest reproduced baseline by 8.5 percentage points while maintaining 99.9% runtime validity.
- Cross-Model Transfer: When the harness evolved for GPT-4o-mini was applied to Qwen3.5-35B-A3B, the ASR dropped from 32.1% to 0.9%, demonstrating that the evolved defenses are not model-specific.
Limitations and Future Work
- Compute Cost: The evolution process is computationally expensive, requiring hundreds of LLM calls for diagnosis, expansion, and evaluation.
- Meta-Model Dependency: The quality of the evolved code depends on the "meta-model" (in this case, GPT-5) used to perform the evolution.
- Fixed Environment: The current framework keeps the tools and environment fixed. Future work could explore evolving the tool definitions themselves to be more inherently secure.
- Single Round of Feedback: The paper primarily explores one full round of stressor-guided feedback. Multiple iterative rounds could potentially push the frontier even further.
Summary updated