TL;DR
This paper argues that evaluating AI agents with a simple "did the attack succeed?" binary is insufficient for real-world deployment. The authors introduce DAMAGEFLOW, a framework that measures the "blast radius"—the actual realized damage—of an agent's failure, and BLASTGUARD, a runtime defense that uses "dry-runs" and least-privilege capability graphs to contain malicious tool calls before they can cause irreversible harm.
What Problem Does This Paper Address?
Current security benchmarks for tool-using AI agents (like those that can send emails or edit code) typically focus on a binary outcome: did the agent follow a malicious instruction or not?
However, in a production environment, not all failures are equal. An agent that drafts a typo-ridden email is a minor nuisance; an agent that deletes a production database or leaks sensitive PII to an external server is a catastrophe. The authors identify a "measurement gap" where two agents might have the identical 20% attack-success rate, yet one causes 89x more actual damage than the other. We lack a standardized way to quantify this "blast radius" and a robust way to prevent it without breaking the agent's utility.
Methods
The paper introduces three core components:
- Blast Radius Score (BRS): A new metric that moves beyond binary success. It calculates damage based on four factors:
- Resource Sensitivity: How "important" is the data being touched?
- Action Impact: Is it a "read" (low impact) or a "delete/send" (high impact)?
- Propagation: Does the action reach external users or downstream systems?
- Recovery Cost: Can the action be undone (like a draft) or is it irreversible (like a sent email)?
- State-Diff Oracles: Instead of using an LLM to "judge" if an agent did something bad, the framework uses deterministic "state-diffs." It looks at the actual changes in the sandbox (e.g., "Was a BCC added to this email?") to provide a mechanically auditable score.
- BLASTGUARD Runtime: A defensive layer that sits between the agent and its tools. It works by:
- Intent-to-Capability Compilation: Turning a user's natural language request into a strict "capability graph" (e.g., "Send Alice the notes" only authorizes
email.sendtoAlice). - Provenance Tracking: Labeling data from untrusted sources (like an incoming email) as "tainted."
- Dry-Runs: Executing high-impact tool calls in a temporary sandbox first to see the "state diff" before committing the action.
Datasets
The authors introduce AGENTBLASTBENCH, a pilot benchmark consisting of 80 deterministic scenarios. These scenarios cover four primary domains:
- Email: Sending notes, handling attachments.
- Docs/RAG: Summarizing policies, publishing documents.
- Repository: Patching files, managing CI/CD workflows.
- MCP Registry: Selecting and using tools via the Model Context Protocol.
The benchmark tests agents against four attack families: indirect prompt injection, tool poisoning, tool shadowing, and multi-agent relay attacks.
Key Insights and Ideas
- The 89x Damage Gap: The most striking insight is that binary metrics hide massive risks. In their pilot, two agents with the same attack-success rate had vastly different BRS scores because one mostly created reversible drafts while the other performed irreversible deletes.
- Auditability over "Vibes": By scoring the committed state (the state-diff) rather than the agent's explanation, the authors provide a security framework that doesn't rely on the "fuzzy" reasoning of an LLM judge.
- Prevented BRS: The authors argue we should measure what a defense stopped. A system with low realized damage is only impressive if you can prove it actually blocked high-risk actions that the agent attempted to take.
- The Utility-Risk Frontier: Security is often a trade-off. The paper formalizes this as "SafeUtility," helping developers find the "sweet spot" where an agent remains helpful but its maximum possible damage is capped.
Key Results
- Effectiveness of BLASTGUARD: In the 80-scenario pilot, BLASTGUARD reduced realized unauthorized damage to zero while retaining 87.5% of the agent's benign utility.
- Failure of Simple Filters: Traditional "PromptGuard" style filters (which look for "bad words" in prompts) were far less effective, as they couldn't catch attacks hidden in tool metadata or non-obvious relay attacks.
- Ablation Success: The authors found that provenance tracking and dry-runs were the most critical components. Removing provenance tracking caused the BRS to jump from 0 to 0.226, as the system lost the ability to see when an agent was using "tainted" data to expand its audience (e.g., adding a hidden BCC from a malicious email).
Limitations and Future Work
- Scale: The pilot is limited to 80 scenarios. The authors acknowledge the need to test this against a wider array of frontier and open-weight models.
- Subjectivity of Weights: The BRS relies on weights for "sensitivity" and "impact." While they provide defaults, different organizations will need to calibrate these to their specific risk tolerance.
- Ambiguity: Compiling a vague user instruction (e.g., "Help me with my emails") into a strict capability graph is difficult. The authors suggest that in these cases, the system should default to a "narrow" scope or ask for clarification.
- Domain Coverage: While email and repos are covered, other common tool environments like Calendars, CRMs, and full Filesystems are specified in the framework but were not fully implemented in the pilot.
Summary updated